I have a few programmable filters that take a long time to execute. I’ve done my best to optimize the single-threaded performance, and now I’m looking at other options for speeding things up. It appears that the Programmable Filter does not take advantage of multiple processes when run locally. Does it have any capability to do this, say if using the client-server workflow? Alternatively, should I expect normal Python tools (i.e. the multiprocessing module) to work within the filter?
@patchett2002 @cory.quammen @utkarsh.ayachit @boonth
If you connect to a server running multiple processes and apply a Programmable Filter, I would think that the filter will run in parallel. Are you able to load the data in parallel, in your situation?
I’ve never tried multiprocessing in a Programmable Filter, but I don’t see why that wouldn’t work. You could also look at the vtkMultiProcessController class in VTK.
If the filter is trivially parallelize-able i.e. it does not involve any reduction and such, you should indeed be simply able to run connect to a parallel pvserver. You can run a parallel-pvserver locally too and directly connect to it from ParaView client.
If it involves reductions or any parallel communication/synchronization, you can use mpi4py or as @boonth suggests, use the multiprocess controller.