Hello,
I am looking to use parallel processing (specifically, Python’s multiprocessing package) to improve computation times of a programmable filter that I have produced. I am wondering if it is at all possible for me to do this?
My filter is computing a value for each cell in my mesh using pre-existing values, unfortunately, the computation time for each cell is fairly long. When scaled to the size of my mesh, the total computation time can reach ~3 days. To alleviate this issue, I was hoping to implement parallel processing (so that I could leverage a multi-core CPU to analyze multiple cells simultaneously).
I started off by building a simple script, with fake data, and testing it in my personal editor (it worked). Below is an extremely simplified code of what I am attempting to do (in reality, there are many nested for-loops inside the function calc_value
, which have been removed for brevity):
import numpy as np
import multiprocessing as mp
# Loading Data
data = inputs[0].CellData['U']
cellCount = inputs[0].GetNumberOfCells()
# Defining Mathematical Manipulation
def ParaTest(i):
uX = data[i][0]
uY = data[i][1]
uZ = data[i][2]
uMag = np.sqrt(uX**2 + uY**2 + uZ**2)
return uMag
# Processing Data in Parallel (with 8 cores)
if __name__ == '__main__':
with mp.Pool(processes=8) as pool:
results = pool.map(ParaTest,range(cellCount))
print(results)
However, when I attempt to run this code I am met with the error:
AttributeError: Can't pickle local object 'RequestData.<locals>.calc_value'
I understand that this error will often arise if the function to be used in the parallel processing algorithm is not defined at the top-level, however, in this case I believe it is?
Any help or pointers would be greatly appreciated!
Edit: I am using Mac OS running on apple silicon.