Improving speed for thresholding of list of values (custom selection+extractSelect)

ParaView Support
1

Hello,
I need to extract cells with specific values of a specific field. For this I am using the following lines of code:

SetActiveSource(filter)
QuerySelect(QueryString=f'(isin({fieldName}, {listOfValues}))', FieldType=FieldType, InsideOut=complementary)
extracted = ExtractSelection(Input=filter,registrationName=registrationName)
SetActiveSource(filter)
ClearSelection()

the listOfValues can be millons therefore this process can be slow, but it works for float, ints and whatever kind of elements (cells or points). I had been trying to improve the speed of this with programmable filters but to whatever i test using numpy and vtk i never get around the same speed as the previous cited lines of code.
So I assume there is no fastest way to do so? If I need to assume that this IS the fastest way (anyone welcome to tell me if there is another fastest way). when doing QuerySelect there is a GUI/visualization part (where the selected elements becomes pink, and then the ClearSelection() will clean it and continue the script, my question is there any way to setoff the visual feedback of the QuerySelect? I am looking to improve further this speed and I execute all the lines at same time, therefore I dont care about ‘showing’ the pink selected cells.

2

hey @Kenichiro-Yoshimi,
sorry for bothering with a direct tag, from the post you made on my other post, I tried implementing something based on it that would use the same logic, but I am not achieving a speed up (actually it takes the double time), would you mind giving me your oppinion?

what I came up to:

def extractSelectionByListOfValues_test(filter=[], listOfValues=[]):
    if filter==[]:
        filter= GetActiveSource()
    if isinstance(filter,str):
        filter=FindSource(filter)
    # create a new 'Programmable Filter'
    programmableFilter1 = ProgrammableFilter(registrationName='ProgrammableFilter1', Input=filter)
    # find source
    # Properties modified on programmableFilter1
    programmableFilter1.Set(
        RequestInformationScript='',
        RequestUpdateExtentScript='',
        PythonPath='',
        Script=f"""import numpy as np
from vtkmodules.vtkCommonCore import vtkIdList
from vtkmodules.vtkCommonDataModel import vtkSelectionNode, vtkSelection
from vtkmodules.vtkFiltersExtraction import vtkExtractSelection
from vtk.numpy_interface import dataset_adapter as dsa
from vtk.util.numpy_support import numpy_to_vtk, vtk_to_numpy
from vtkmodules.util.numpy_support import numpy_to_vtkIdTypeArray
# --- Get inputs ---
def get_vtk_dataset(wrapped_input):
    vtk_obj = wrapped_input.VTKObject
    if vtk_obj.IsA("vtkMultiBlockDataSet"):
        return vtk_obj.GetBlock(0)
    return vtk_obj

typeOfIDs='CellIds'

#get the input object
vtk_ref = get_vtk_dataset(inputs[0])   # removedLayer
wrap_ref = dsa.WrapDataObject(vtk_ref)
#get the list of IDs in a np.array for typeOfIDs
gid_ref = np.asarray(wrap_ref.CellData[typeOfIDs]).ravel() #typeOfIDs="CellIds"
cell_array = np.asarray(wrap_ref.CellData["CellIds"])
listOfValues={listOfValues}
if listOfValues:
    if isinstance(listOfValues,list):
        gid_toExtract=np.array(listOfValues)
    elif isinstance(listOfValues,numpy.ndarray):
        gid_toExtract=listOfValues
    else:
        error

else:
    vtk_tgt = get_vtk_dataset(inputs[1])   # finalChannel
    nCells = vtk_tgt.GetNumberOfCells()
    wrap_tgt = dsa.WrapDataObject(vtk_tgt)
    gid_toExtract = np.asarray(wrap_tgt.CellData[typeOfIDs]).ravel() #typeOfIDs="CellIds"

out = self.GetOutputDataObject(0)

# Sort ref, then searchsorted to find which target points exist in ref
gid_ref_sorted = np.sort(gid_ref)


pos = np.searchsorted(gid_ref_sorted, gid_toExtract)
pos_clipped = np.clip(pos, 0, len(gid_ref_sorted) - 1)
mask = gid_ref_sorted[pos_clipped] == gid_toExtract
shared_local_ids = np.nonzero(mask)[0]


selection_ids = numpy_to_vtkIdTypeArray(pos_clipped[shared_local_ids].astype(np.int64), deep=1)

node = vtkSelectionNode()
node.SetFieldType(vtkSelectionNode.CELL)
node.SetContentType(vtkSelectionNode.INDICES)
node.SetSelectionList(selection_ids)

sel = vtkSelection()
sel.AddNode(node)

extract = vtkExtractSelection()
extract.SetInputDataObject(0, vtk_ref)
extract.SetInputDataObject(1, sel)
extract.Update()

out.ShallowCopy(extract.GetOutputDataObject(0))
""",
    )
    return programmableFilter1

vs

def extractSelectionByListOfValues_old(filter=[], listOfValues=[]):
    SetActiveSource(filter)
    QuerySelect(QueryString=f'(isin({fieldName}, {listOfValues}))', FieldType=FieldType, InsideOut=complementary)
    extracted = ExtractSelection(Input=filter,registrationName=registrationName)
    SetActiveSource(filter)
    ClearSelection()
return extracted

I was hopping a speedup due to paraview creating the ‘pink cells’ when using QuerySelect, but for my current test, extractSelectionByListOfValues_old is running in 14 s, where extractSelectionByListOfValues_test is taking 32 s.
the filter is a unstructured grid of 630 209 cells and for the test I using the function several times for different lengths of listOfValues ([1766, 367694, 2, 86990, 4, 27714, 20387, 11144, 26251, 8742, 27682, 10872, 18895, 19929, 1642, 4, 5, 4, 480, 2] cells each test) so rader small, as in your test case the speed was crazy even for larger examples, i am afraid i am missing something.

thanks (and sorry for the tag, but since I was basing the funciton on your post, would appreciate your oppinion)

3

Hi @otaolafr,

A couple of things explain what you’re seeing:

Your 20 lengths sum to exactly 630,209 — you’re partitioning the entire grid into 20 groups. So the per-call overhead is paid 20×, and that’s what dominates — not the extraction itself.

That per-call overhead is two things your _test adds and QuerySelect doesn’t:

  1. Creating a fresh ProgrammableFilter every call.
  2. Compiling the value list into the script string (listOfValues={listOfValues}). For the 367,694-value call, just compile()-ing that literal is ~0.85 s — paid every time.

I reproduced both paths with plain VTK (same vtkExtractSelection engine), and the extraction kernel itself is sub-second even for the large lists — so the bottleneck was never the extraction, it’s the per-call pipeline-creation + source-compilation overhead. You’re not missing anything.

So: don’t bake the values into the script (pass them as real data), and reuse one filter instead of rebuilding it 20×.
Also note your searchsorted→INDICES step is actually incorrect unless CellIds is already 0..N-1 (it feeds sorted positions as original indices); np.isin(gid_ref, values) → np.nonzero(…)[0] gives the correct original indices and was the fastest path in my test.

4

Thanks for your answer and taking the time.
my issue is that i need to do different things with each of the 20 groups, for each of them i will apply another function that will do a lot of checks and extract its external surface, so I need to be able to differentiate them.

So maybe create a mask, so each of the 20s have a different value and after filter using the mask, to get ‘separate’ intermediate outputs where i will call the function? do you think there would be a big improvement by doing this? I am asking as i am jumping a little bit to the void, not sure that it will be outperform and i will encounter a lot of bottle necks as i dont know vtk library to the level of doing what i am doing in paraview with the filters themselves…

  1. Compiling the value list into the script string (listOfValues={listOfValues}). For the 367,694-value call, just compile()-ing that literal is ~0.85 s — paid every time.

if i pass a filter and recover its values instead as i do with vtk_tgt i should get a speed up then? but not so much as the 367694 is the largest one by far, therefore the other ones wouls be less than 0.85 s so the speed up should be >>20s, no?

Also note your searchsorted→INDICES step is actually incorrect unless CellIds is already 0..N-1 (it feeds sorted positions as original indices); np.isin(gid_ref, values) → np.nonzero(…)[0] gives the correct original indices and was the fastest path in my test.

the cellIds beging in 0, but yes you are right, this part on your original code, was a little bit confusing for me to follow up. thanks again.

EDIT: also i just noticed that in pvbatch the overhead is crazy reduced! which is quite middle ground as i can check in paraview and if it works then scale up in pvpython. still appreciate your opinion but just posting it in case of others need.