Parallel Processing in Programmable Filter

Hello,

I am looking to use parallel processing (specifically, Python’s multiprocessing package) to improve computation times of a programmable filter that I have produced. I am wondering if it is at all possible for me to do this?

My filter is computing a value for each cell in my mesh using pre-existing values, unfortunately, the computation time for each cell is fairly long. When scaled to the size of my mesh, the total computation time can reach ~3 days. To alleviate this issue, I was hoping to implement parallel processing (so that I could leverage a multi-core CPU to analyze multiple cells simultaneously).

I started off by building a simple script, with fake data, and testing it in my personal editor (it worked). Below is an extremely simplified code of what I am attempting to do (in reality, there are many nested for-loops inside the function calc_value, which have been removed for brevity):

import numpy as np
import multiprocessing as mp

# Loading Data
data = inputs[0].CellData['U']
cellCount = inputs[0].GetNumberOfCells()

# Defining Mathematical Manipulation
def ParaTest(i):
    uX = data[i][0]
    uY = data[i][1]
    uZ = data[i][2]
    uMag = np.sqrt(uX**2 + uY**2 + uZ**2)

    return uMag

# Processing Data in Parallel (with 8 cores)
if __name__ == '__main__':
    with mp.Pool(processes=8) as pool:
        results = pool.map(ParaTest,range(cellCount))
        print(results)

However, when I attempt to run this code I am met with the error:

AttributeError: Can't pickle local object 'RequestData.<locals>.calc_value'

I understand that this error will often arise if the function to be used in the parallel processing algorithm is not defined at the top-level, however, in this case I believe it is?

Any help or pointers would be greatly appreciated!

Edit: I am using Mac OS running on apple silicon.

@cory.quammen @mwestphal

Pickle suport has been added recently to VTK and ParaView, so maybe that fixes it for you. You may want to try the last nightly.

FYI @Charles_Gueunet @jfausty

To give a bit more details, we should be able to pickle some vtkDataObject in Paraview starting with this version.
The list of the supported object is available in the MR: here
I hope it will be enough for your case.

Thank you @mwestphal and Charles , I will upgrade my Paraview software to the most recent nightly release (I am currently running 5.11.0). I will let you know how things go.

If I am unable to get the multiprocessing toolbox to work, would I be able use VTK’s vtkMultiProcessController to parallelize my programmable filter?

Edit: I attempted to implement VTK’s vtkMultiProcessController, however, I am only able to access a single core (rather than the 10 cores available on my machine). Is there a way grant the filter access to my other cores? Or this issue related to the client-server aspect of Paraview?

Hello @Charles_Gueunet,

I am a fairly novice programmer so forgive me if my questions are rudimentary. Will I even be able to use the pickle support that has been added to Paraview? Because the pickling that is done by the multiprocessing library is embedded within that module?

I have downloaded the most recent nightly version, however, I am unsure of how I can implement the pickle support into my code above.