possibility to use multithread/multiprocessor ?

ParaView Support
1

Hello,
I have a list of filters which are independent one from the other, and right now I am appliying a function to each of this filter sequentially. The function itself, uses some paraview filters BUT at the end of the function it will do some calculations, delete all the filters that were generated during the function itself and return a value, so at the end of the application of the function to each element of the list I get a value for each element, and the pipeline stays the same (as the created in the middle filters were deleted.).
The function uses some filters that are not parallelized by paraview (it is using two, clip and connectivity). as the list of inputs is quite large, I would like to use the different processors or threads on my machine, to do all (or at least a maximum possible number depending on the hardware) calls of this function in parallel. is this something possible to do?
here i give a small clean example for easy coomprehension:

from paraview.simple import *
import gc
from paraview.vtk.numpy_interface import dataset_adapter as dsa

def myFunction(filter):
    slice1 = Slice(registrationName='Slice1', Input=filter)
    connectivity1=Connectivity(registrationName='Connectivity1', Input=slice1)
    UpdatePipeline()
    data=dsa.WrapDataObject(servermanager.Fetch(connectivity1))
    vals=data.CellData['RegionId'].tolist()
    if vals==[] or vals==0:
        flag=0
    else:
        flag=len(set(vals))
    for f in [slice1,connectivity1]:
        Delete(f)
        del f
    gc.collect()
    return flag

wavelet1 = Wavelet(registrationName='Wavelet1')
UpdatePipeline()
pointDatatoCellData1 = PointDatatoCellData(registrationName='PointDatatoCellData1', Input=wavelet1)

lowers=[  100, 150,200,250]
uppers=[ 150, 200,250,300]
thresholds=[Threshold(registrationName='Threshold1', Input=pointDatatoCellData1,Scalars = ['CELLS', 'RTData'],LowerThreshold=lowers[n], UpperThreshold=uppers[n]) for n,i in enumerate(lowers)]

thresholdsf=[myFunction(filt) for filt in thresholds]

would it be possible to do the thresholdsf but with multithread/multiprocessor?
I have on the workstation 112 threads and i see my code running in a single one…

2

Clip is multithreaded. Connectivity indeed is not.

I would like to use the different processors or threads on my machine, to do all (or at least a maximum possible number depending on the hardware) calls of this function in parallel. is this something possible to do?

Use distributed data processing using MPI. 8. Remote and parallel visualization — ParaView Documentation 6.1.0 documentation

3

Clip is multithreaded. Connectivity indeed is not.

can understand why connectivity is not, but maybe i am missexplaining myself. what i would like to do is do multiprocessing/threading in objects that have the same source but they are independant one from the other.

as in the example i gave, i have a filter where i extract specific cells using a list of IDs from original source, and a second filter that i also extracted from original source with a different list of IDs.
i want to do some checking for each of them at the same time, like, applying countour (just an example) to the two at the same time outside of the countour being multithreaded or not. as they are different datas, something like running two instances of paraview and doing the same on each of them but for different data. the only difference with what i just mentionned is that, as the two sources i want to apply this filters (ie., in this example, countour) are comming from a same original source, therefore they are the two of them in same instance of paraview.
i dont know if i example mayself better with this comment. please let me know if it is not the case.

so in the original post, be able to only do this line:

thresholdsf=[myFunction(filt) for filt in thresholds]

but in parallel, so each processor gets its own element (ie., filt) does the calculation in ‘serial (or parallel depending on the internal filters of myFunction) inside the processor’ and returns back the result (which is a int).

4

This is distributed processing :slight_smile:

5

this is faisable in a python script?
i mean, take the first script of my post, and be able to ‘distribute processing’ the last line:

thresholdsf=[myFunction(filt) for filt in thresholds]

so each processor/thread runs the myFunction at same time, and after one can assamble thresholdsf list? (if you look at the myFunction, it returns only an int)

6

Yes, using pvbatch . But its not exactly what you are talking about.

Please read the doc I shared above.

7

Hello, mathieu,
I have read the doc you post, if I understand correctly, I should run,
mpirun -np 4 pvserver
and then run:
mpirun -np 4 pvbatch script.py

but correct me if i am wrong,
but this will basically, divide my input data into 4 and treat each of the quarters separatly (more or less), no?

if I am correct with my previous question, this would not be helpful for my current issue, as what i require is that the way i divide the data, is driven by calculations i do previously (inside paraview), and once the calculations are done, i want to separate the data by these results.
for example, if i want to divide a wavelet filter, with this, it will divide it as in image 8.17 into 4 exact similar quarters (of your link), but instead i would like to divide the data by lets say different values of thresholds, such as lets say: with values of RTData between 20-100, values between 100-150, 150-200 and 200-280.
this will divide my data in 4 different sections:

image
image519×535 82.3 KB

is it possible to apply the function to each of the sections at same time?
the separation is more complex than simply applying a threshold with different values, but it is just to better exemplify my issue. each of the sections(outputs of the thresholds) are ‘independent’ to the other but at same time coming from same original data.

furthermore, i dont know with anticipation how many ‘sections’ i will have. what i am thinking is something more like, having a pool of filters, that a processor will pick up when it is free, do the calculation, return the value and after pick another of the sections if there is remaining ones.

from what i understand on the doc this is not faisable, as when i run pvbatch, paraview will divide the data as in image 8.17 and work on each block more or less ‘separatly’, no?
lastly, I see that you are active also on the support of the vtk for my current case (reason of this post) if it is not feasable what i require in paraview, would this be a possible solution? or would i find myself with same bottle neck? i am asking as i imagine that it is an issue of the paraview proxy? maybe not and i should look for another library such as scikit?

I would appreciate your input on this subject, as my current code works and does everything i expect to, but it takes too much time, as i can not parallelize this it is a big issue.