Fix Python – Multiprocessing a for loop?


Asked By – ChrisFro

I have an array (called data_inputs) containing the names of hundreds of astronomy images files. These images are then manipulated. My code works and takes a few seconds to process each image. However, it can only do one image at a time because I’m running the array through a for loop:

for name in data_inputs:'.fits')
    #image is manipulated

There is no reason why I have to modify an image before any other, so is it possible to utilise all 4 cores on my machine with each core running through the for loop on a different image?

I’ve read about the multiprocessing module but I’m unsure how to implement it in my case.
I’m keen to get multiprocessing to work because eventually I’ll have to run this on 10,000+ images.

Now we will see solution for issue: Multiprocessing a for loop?


You can simply use multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):'{}.fits'.format(name))

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool, data_inputs)  # process data_inputs iterable with pool

This question is answered By – alko

This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0