possibility to use multithread/multiprocessor ?

ParaView Support
1

Hello,
I have a list of filters which are independent one from the other, and right now I am appliying a function to each of this filter sequentially. The function itself, uses some paraview filters BUT at the end of the function it will do some calculations, delete all the filters that were generated during the function itself and return a value, so at the end of the application of the function to each element of the list I get a value for each element, and the pipeline stays the same (as the created in the middle filters were deleted.).
The function uses some filters that are not parallelized by paraview (it is using two, clip and connectivity). as the list of inputs is quite large, I would like to use the different processors or threads on my machine, to do all (or at least a maximum possible number depending on the hardware) calls of this function in parallel. is this something possible to do?
here i give a small clean example for easy coomprehension:

from paraview.simple import *
import gc
from paraview.vtk.numpy_interface import dataset_adapter as dsa

def myFunction(filter):
    slice1 = Slice(registrationName='Slice1', Input=filter)
    connectivity1=Connectivity(registrationName='Connectivity1', Input=slice1)
    UpdatePipeline()
    data=dsa.WrapDataObject(servermanager.Fetch(connectivity1))
    vals=data.CellData['RegionId'].tolist()
    if vals==[] or vals==0:
        flag=0
    else:
        flag=len(set(vals))
    for f in [slice1,connectivity1]:
        Delete(f)
        del f
    gc.collect()
    return flag

wavelet1 = Wavelet(registrationName='Wavelet1')
UpdatePipeline()
pointDatatoCellData1 = PointDatatoCellData(registrationName='PointDatatoCellData1', Input=wavelet1)

lowers=[  100, 150,200,250]
uppers=[ 150, 200,250,300]
thresholds=[Threshold(registrationName='Threshold1', Input=pointDatatoCellData1,Scalars = ['CELLS', 'RTData'],LowerThreshold=lowers[n], UpperThreshold=uppers[n]) for n,i in enumerate(lowers)]

thresholdsf=[myFunction(filt) for filt in thresholds]

would it be possible to do the thresholdsf but with multithread/multiprocessor?
I have on the workstation 112 threads and i see my code running in a single one…

2

Clip is multithreaded. Connectivity indeed is not.

I would like to use the different processors or threads on my machine, to do all (or at least a maximum possible number depending on the hardware) calls of this function in parallel. is this something possible to do?

Use distributed data processing using MPI. 8. Remote and parallel visualization — ParaView Documentation 6.1.0 documentation

3

Clip is multithreaded. Connectivity indeed is not.

can understand why connectivity is not, but maybe i am missexplaining myself. what i would like to do is do multiprocessing/threading in objects that have the same source but they are independant one from the other.

as in the example i gave, i have a filter where i extract specific cells using a list of IDs from original source, and a second filter that i also extracted from original source with a different list of IDs.
i want to do some checking for each of them at the same time, like, applying countour (just an example) to the two at the same time outside of the countour being multithreaded or not. as they are different datas, something like running two instances of paraview and doing the same on each of them but for different data. the only difference with what i just mentionned is that, as the two sources i want to apply this filters (ie., in this example, countour) are comming from a same original source, therefore they are the two of them in same instance of paraview.
i dont know if i example mayself better with this comment. please let me know if it is not the case.

so in the original post, be able to only do this line:

thresholdsf=[myFunction(filt) for filt in thresholds]

but in parallel, so each processor gets its own element (ie., filt) does the calculation in ‘serial (or parallel depending on the internal filters of myFunction) inside the processor’ and returns back the result (which is a int).

4

This is distributed processing :slight_smile:

5

this is faisable in a python script?
i mean, take the first script of my post, and be able to ‘distribute processing’ the last line:

thresholdsf=[myFunction(filt) for filt in thresholds]

so each processor/thread runs the myFunction at same time, and after one can assamble thresholdsf list? (if you look at the myFunction, it returns only an int)

6

Yes, using pvbatch . But its not exactly what you are talking about.

Please read the doc I shared above.

7

Hello, mathieu,
I have read the doc you post, if I understand correctly, I should run,
mpirun -np 4 pvserver
and then run:
mpirun -np 4 pvbatch script.py

but correct me if i am wrong,
but this will basically, divide my input data into 4 and treat each of the quarters separatly (more or less), no?

if I am correct with my previous question, this would not be helpful for my current issue, as what i require is that the way i divide the data, is driven by calculations i do previously (inside paraview), and once the calculations are done, i want to separate the data by these results.
for example, if i want to divide a wavelet filter, with this, it will divide it as in image 8.17 into 4 exact similar quarters (of your link), but instead i would like to divide the data by lets say different values of thresholds, such as lets say: with values of RTData between 20-100, values between 100-150, 150-200 and 200-280.
this will divide my data in 4 different sections:

image
image519×535 82.3 KB

is it possible to apply the function to each of the sections at same time?
the separation is more complex than simply applying a threshold with different values, but it is just to better exemplify my issue. each of the sections(outputs of the thresholds) are ‘independent’ to the other but at same time coming from same original data.

furthermore, i dont know with anticipation how many ‘sections’ i will have. what i am thinking is something more like, having a pool of filters, that a processor will pick up when it is free, do the calculation, return the value and after pick another of the sections if there is remaining ones.

from what i understand on the doc this is not faisable, as when i run pvbatch, paraview will divide the data as in image 8.17 and work on each block more or less ‘separatly’, no?
lastly, I see that you are active also on the support of the vtk for my current case (reason of this post) if it is not feasable what i require in paraview, would this be a possible solution? or would i find myself with same bottle neck? i am asking as i imagine that it is an issue of the paraview proxy? maybe not and i should look for another library such as scikit?

I would appreciate your input on this subject, as my current code works and does everything i expect to, but it takes too much time, as i can not parallelize this it is a big issue.

8

also, just in case I tested doing:

Programs/ParaView/ParaView-6.0.1-MPI-Linux-Python3.12-x86_64/bin/mpiexec     -np 4     Programs/ParaView/ParaView-6.0.1-MPI-Linux-Python3.12-x86_64/bin/pvserver

Programs/ParaView/ParaView-6.0.1-MPI-Linux-Python3.12-x86_64/bin/mpiexec     -np 4     Programs/ParaView/ParaView-6.0.1-MPI-Linux-Python3.12-x86_64/bin/pvbatch Downloads/Scripts/scriptPy.py

but I am getting an ouput equal to VTKNoneArray where in the ‘non parallel version’ (without modification of the script) i am getting correct results:

    N=len(set(data.PointData['RegionId']))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'VTKNoneArray'

without creating the server and without running pvbatch thought mpiexec I do not get this error and I get a value for N.

9

Wrong, you should only run mpirun -np 4 pvbatch script.py

this will basically, divide my input data into 4 and treat each of the quarters separatly

Yes

with values of RTData between 20-100, values between 100-150, 150-200 and 200-280.
this will divide my data in 4 different sections:

You can do that by writing a C++ filter that would do this data distribution but it will be unstructured grid instead of image data and will require extensive communications between ranks, I dont think you want that tbh.

having a pool of filters, that a processor will pick up when it is free, do the calculation, return the value and after pick another of the sections if there is remaining ones.

This is rough grain task based parallelism, not supported by VTK/ParaView

In any case you may need help from Kitware to help you with your script.

10

Hello mathieu,
thanks for the answer and sorry for the late reply.

You can do that by writing a C++ filter that would do this data distribution but it will be unstructured grid instead of image data and will require extensive communications between ranks, I dont think you want that tbh.

I agree with you that it is not what i am looking for, as in this case it would require communication in between where in my case they are independent, they only have the same input but they are basically as if i would save each of the information in separate files and then read them.

This is rough grain task based parallelism, not supported by VTK/ParaView

before holydays, i tested roughtly the same script that i post it in first comment but translated to vtk library, and i was able to run them in parallel by multithreading. the funciton was only doing a clip, not doing conectivity, but maybe this might work? i will keep you post it. but at first glance i was able to do ‘more’ with vtk itself, than with paraview (i need to validate that i am able to do ‘all’)

In any case you may need help from Kitware to help you with your script.

sadly i dont have a budged assigned that i could pay for custom help, if there is any other way around (like addind what eventually come up from the work to paraview itself) i am more than open, but sadly i dont have a funding to pay directly outside of my own ‘men hours’. if there is any possibility of something like this, please do not hesitate.

regards.

11

Most vtk filter supports multithreading and you definitely should use it. Im not sure what is your question though.

12

Sorry for not being clear,
the thing is, as you notice from the original post, I want to execute a function that will create, do some calculations to get a int value, and then delete the filters created on the funciton.
As i need to apply this function to a bunch of different filters that are at same level on the tree structure of the pipeline (described in this comment). As in some cases the function takes a a few minuts where other times it takes a few seconds (due to calculation depending on the size of each of this input filters) AND that the filters are at the same level on the three (i mean it is not like one is created from another and another one from the later (not nested). I thought that i could apply one of the concurrent python library so i create a pool of all this filters, and apply the function in multithreading/multiprocessing to it, so i can do distributed processing, as i noticed that my processors (using htop) where almost all free, and while ‘waiting’ for the function to end in a specific filter, the other processors could hop and continue with the other ones so they are done at same time.
for this I tried in the beigning was to use threadPoolExecutor, from concurrent python library. this in paraview, did not worked nor multiprocessing, i imagine that the proxys of paraview layer does not help on this. but when i did it in vtk, it looked like it was working, so in vtk i did:

from concurrent.futures import ThreadPoolExecutor
max_workers=16
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    thresholdsf = list(executor.map(myFunction, thresholds))

where the thresholds was a list of vtk.vtkThreshold() setted to different values etc etc, and in this case it runned correctly. but i have never used vtk library alone before this, and i must confess that i would love to stay on paraview side as it is something i know.

Most vtk filter supports multithreading and you definitely should use it. Im not sure what is your question though.

the function itself consist of a bunch of slices and connectivity filters used several times, as you mentionned connectivity does not support multithreading. what i want to do is do in serial all the filters of the function to a bunch of different inputs in parallel.

sadly i dont have a budged assigned that i could pay for custom help, if there is any other way around (like addind what eventually come up from the work to paraview itself) i am more than open, but sadly i dont have a funding to pay directly outside of my own ‘men hours’. if there is any possibility of something like this, please do not hesitate.

in this regards, i not sure if its possible.

13

I do not think this is possible or even advisable to use this approach in the context of ParaView.

14

and do you see any possible solution while staying inside paraview?
the thing is that the list of ‘thresholds’ that i need to run my function on, are a few thousands and sometimes it literally takes 15 min, and most of the time a few secs, i tried to optimized the function itself to squish every drop of time that i can, but it still long sometimes.

the current implementation of paraview of parallelism that you pointed to me before it would not help

and be able to do something like i was asking for (run them in a pool) would allow a proc to take the 15 min one, while the other ones remove the really small ones.

15

No. Maybe this is something that ParaView should be able to do, to have a filter level parallelism but the effort seems monumental to add yet another way to parallel compute.