# Parallel Processing in Programmable Filter

**URL:** https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342
**Category:** ParaView Support
**Created:** [February 6, 2023, 8:16pm UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342 "2023-02-06T20:16:12Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Matthew](https://discourse.paraview.org/letter_avatar_proxy/v4/letter/m/ed655f/32.png) [@Matthew](https://discourse.paraview.org/u/Matthew)
#### Post date: [February 6, 2023, 8:16pm UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342/1 "2023-02-06T20:16:12Z")

</div>

Hello,

I am looking to use parallel processing (specifically, Python’s multiprocessing package) to improve computation times of a programmable filter that I have produced. I am wondering if it is at all possible for me to do this?

My filter is computing a value for each cell in my mesh using pre-existing values, unfortunately, the computation time for each cell is fairly long. When scaled to the size of my mesh, the total computation time can reach ~3 days. To alleviate this issue, I was hoping to implement parallel processing (so that I could leverage a multi-core CPU to analyze multiple cells simultaneously).

I started off by building a simple script, with fake data, and testing it in my personal editor (it worked). Below is an extremely simplified code of what I am attempting to do (in reality, there are many nested for-loops inside the function `calc_value`, which have been removed for brevity):

```auto
import numpy as np
import multiprocessing as mp

# Loading Data
data = inputs[0].CellData['U']
cellCount = inputs[0].GetNumberOfCells()

# Defining Mathematical Manipulation
def ParaTest(i):
    uX = data[i][0]
    uY = data[i][1]
    uZ = data[i][2]
    uMag = np.sqrt(uX **2 + uY** 2 + uZ**2)

    return uMag

# Processing Data in Parallel (with 8 cores)
if __name__ == ' __main__':
    with mp.Pool(processes=8) as pool:
        results = pool.map(ParaTest,range(cellCount))
        print(results)

```

However, when I attempt to run this code I am met with the error:

```auto
AttributeError: Can't pickle local object 'RequestData.<locals>.calc_value'

```

I understand that this error will often arise if the function to be used in the parallel processing algorithm is not defined at the top-level, however, in this case I believe it is?

Any help or pointers would be greatly appreciated!

Edit: I am using Mac OS running on apple silicon.

@cory.quammen @mwestphal

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.paraview.org/user_avatar/discourse.paraview.org/mwestphal/32/17_2.png) [@mwestphal](https://discourse.paraview.org/u/mwestphal)
#### Post date: [February 7, 2023, 8:54am UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342/2 "2023-02-07T08:54:49Z")

</div>

> [@Matthew](#):
>
> `Can't pickle local object 'RequestData.<locals>.calc_value'`

Pickle suport has been added recently to VTK and ParaView, so maybe that fixes it for you. You may want to try the last nightly.

FYI @Charles_Gueunet @jfausty

---

<div class="post-metadata">

### Author: ![Charles\_Gueunet](https://discourse.paraview.org/user_avatar/discourse.paraview.org/charles_gueunet/32/40_2.png) [@Charles\_Gueunet](https://discourse.paraview.org/u/Charles_Gueunet)
#### Post date: [February 7, 2023, 4:09pm UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342/3 "2023-02-07T16:09:18Z")

</div>

To give a bit more details, we should be able to pickle some `vtkDataObject` in Paraview starting with [this version](https://gitlab.kitware.com/paraview/paraview/-/commit/a60f84533afbe1cbb4e6c9e94052317f70f6b7e0).  
The list of the supported object is available in the MR: [here](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/9862/diffs#5ad96e7d19511d4f4c251b99ef2a1a8c661e8463)  
I hope it will be enough for your case.

---

<div class="post-metadata">

### Author: ![Matthew](https://discourse.paraview.org/letter_avatar_proxy/v4/letter/m/ed655f/32.png) [@Matthew](https://discourse.paraview.org/u/Matthew)
#### Post date: [February 7, 2023, 5:43pm UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342/4 "2023-02-07T17:43:51Z")

</div>

Thank you @mwestphal and Charles , I will upgrade my Paraview software to the most recent nightly release (I am currently running 5.11.0). I will let you know how things go.

If I am unable to get the multiprocessing toolbox to work, would I be able use VTK’s vtkMultiProcessController to parallelize my programmable filter?

**Edit:** I attempted to implement VTK’s vtkMultiProcessController, however, I am only able to access a single core (rather than the 10 cores available on my machine). Is there a way grant the filter access to my other cores? Or this issue related to the client-server aspect of Paraview?

---

<div class="post-metadata">

### Author: ![Matthew](https://discourse.paraview.org/letter_avatar_proxy/v4/letter/m/ed655f/32.png) [@Matthew](https://discourse.paraview.org/u/Matthew)
#### Post date: [February 7, 2023, 7:25pm UTC](https://discourse.paraview.org/t/parallel-processing-in-programmable-filter/11342/5 "2023-02-07T19:25:47Z")

</div>

Hello @Charles_Gueunet,

I am a fairly novice programmer so forgive me if my questions are rudimentary. Will I even be able to use the pickle support that has been added to Paraview? Because the pickling that is done by the multiprocessing library is embedded within that module?

I have downloaded the most recent nightly version, however, I am unsure of how I can implement the pickle support into my code above.