[Python] Multiprocess with multiple arguments

To run the program in different cores in parallel, we use the library multiprocessing to spawn a pool of processes, and map the function to the processes. A simple problem with a single argument is as follows

from multiprocessing.pool import Pool def my_fun(x):    return 2*x input_list = [3,4,5] num_of_processes = 4 Pool pool = Pool(num_of_processes) result_list = pool.map(my_fun, input_list) # the second argument has to be a list pool.close() pool.join() print result_list # should be [6,8,10]

It is simple and easy to understand: we set up how many processes we want, fit the function and input data to the processes.

What if I want to send one more argument into the function? Say, add a constant to the output (and the constant remain the same for different input x)

def my_fun2(x, general_const): return 2*x + general_const

You can simply wrap the arguments if you wish. But I will introduce a fancier way to do this: use partial. See the code below

from multiprocessing.pool import Pool from functools import partial def my_fun2(x, general_const): return 2*x + general_const input_list = [3,4,5] my_const = 1000 num_of_processes = 4 Pool pool = Pool(num_of_processes) result_list = pool.map(partial(my_fun2, general_const=my_const), input_list) ## if you prefer, you can also separate them (just another layout, does not change anything) # my_fun2_partial = partial(my_fun2, general_const=my_const) # result_list = pool.map(my_func2_partial, input_list) pool.close() pool.join() print result_list # should be [1006,1008,1010]

That's it!