I am looking to use parallel processing (specifically, Python’s multiprocessing package) to improve computation times of a programmable filter that I have produced. I am wondering if it is at all possible for me to do this?
My filter is computing a value for each cell in my mesh using pre-existing values, unfortunately, the computation time for each cell is fairly long. When scaled to the size of my mesh, the total computation time can reach ~3 days. To alleviate this issue, I was hoping to implement parallel processing (so that I could leverage a multi-core CPU to analyze multiple cells simultaneously).
I started off by building a simple script, with fake data, and testing it in my personal editor (it worked). Below is an extremely simplified code of what I am attempting to do (in reality, there are many nested for-loops inside the function
calc_value, which have been removed for brevity):
import numpy as np import multiprocessing as mp # Loading Data data = inputs.CellData['U'] cellCount = inputs.GetNumberOfCells() # Defining Mathematical Manipulation def ParaTest(i): uX = data[i] uY = data[i] uZ = data[i] uMag = np.sqrt(uX**2 + uY**2 + uZ**2) return uMag # Processing Data in Parallel (with 8 cores) if __name__ == '__main__': with mp.Pool(processes=8) as pool: results = pool.map(ParaTest,range(cellCount)) print(results)
However, when I attempt to run this code I am met with the error:
AttributeError: Can't pickle local object 'RequestData.<locals>.calc_value'
I understand that this error will often arise if the function to be used in the parallel processing algorithm is not defined at the top-level, however, in this case I believe it is?
Any help or pointers would be greatly appreciated!
Edit: I am using Mac OS running on apple silicon.