Can vtkDataArraySelection be used with custom plug-in filter that is based on the VTKPythonAlgorithmBase approach?

I am trying to write a ParaView Python custom plug-in filter that appends to data in the ParaView pipeline with data that is read in from a file. Since files can have multiple datasets, I am hoping to allow users to select which datasets to load in before applying the filter.

Currently, I am only able to expose the data array selector after hitting apply in ParaView. Is there a way to expose the data array selector upon selecting a file name (before hitting apply)? My current code is shown below. This filter is based on the reader example provided in https://github.com/Kitware/ParaView/blob/master/Examples/Plugins/PythonAlgorithm/PythonAlgorithmExamples.py

The python class for the filter is

@smproxy.filter(label="AppendFromFile")
@smproperty.input(name="Input")
class AppendFromCSV(VTKPythonAlgorithmBase):
  def __init__(self):
    VTKPythonAlgorithmBase.__init__(self, 
                                    nInputPorts=1, 
                                    nOutputPorts=1, 
                                    outputType="vtkUnstructuredGrid")
    self._filename = None
    self._ndata = None
    self._arrayselection = vtkDataArraySelection()
    self._arrayselection.AddObserver("ModifiedEvent",
                                     createModifiedCallBack(self))

The file name is specified through the GUI using the following method. Note that the self._arrayselection.AddArray is called here because I want the data array selector to be available upon filename selection.

@smproperty.stringvector(name="FileName")
@smdomain.filelist()
@smhint.filechooser(extensions="csv", file_description="CSV files")
def SetFileName(self, name):
  if self._filename != name:
    self._filename = name
    # populate dataset names in data array selector.
    if self._filename != 'None':
      with open(self._filename, 'r') as f:
        reader = csv.reader(f)
        dataset_names = reader.next()
      for dname in dataset_names:
        self._arrayselection.AddArray(dname)
    self._ndata = None
    self.Modified()

The data array selector is exposed using

@smproperty.dataarrayselection(name="data values")
def GetDataArraySelection(self):
  return self._arrayselection

Finally, the request data method is

def RequestData(self, request, inInfo, outInfo):
  # get handle on input.
  input0 = dsa.WrapDataObject(vtkUnstructuredGrid.GetData(inInfo[0]))
  # get number of cells
  num_cells = input0.GetNumberOfCells()
  # read in data from file
  raw_data = numpy.getfromtxt(self._filename, delimiter=',', names=True)
  # get handle on output
  output = dsa.WrapDataObject(vtkUnstructuredGrid.GetData(outInfo))
  output.ShallowCopy(vtkUnstructuredGrid.GetData(inInfo[0]))

  # Only add data that is selected in data array selector
  for id, name in enumerate(raw_data.dtype.names):
    if self._arrayselection.ArrayIsEnabled(name):
      data = numpy.zeros([num_cells, ], dtype=float)
      for cell in range(num_cells):
        data[cell] = raw_data[cell][id]
      output.CellData.append(data, name)
  return 1

I think you can populate self._arrayselection in the method RequestInformation.

That method is called before the Apply button is clicked.

HTH,

Loic

In order to populate self._arrayselection, the self._filename has to have the correct path. However, in the implementation above, the path for self._filename is set after hitting apply.

Is there a way to specify self._filename before hitting apply?

Thanks,
Victor

You have to declare your filter as a reader. There is an example in the link you provided.

Hi Mathieu,
I tried switching the decorator on the constructor from

@smproxy.filter(label="AppendFromFile")

to

@smproxy.reader(name="AppendFromCSV(VTKPythonAlgorithmBase", 
                label="Append data from CSV",
                extensions="csv",
                file_description="CSV files") 

and the behavior doesn’t really change. I still do not see a clear way to set the self._filename variable before hitting the apply. And without having a handle on the filename, I am unable to populate self._arrayselection with the appropriate list of data sets that a user can choose from before hitting apply.

Thank you,
Victor

Alternatively, is there a way for the users to provide a string as a separate parameter (outside of the SetFileName) method before hitting apply?
Thanks,
Victor

Actually, you have to separate your filter in two parts.

First parts is a reader that reads the .csv file, you may be able to use the already existing csv reader for that.
Second part is your “append” filter that takes two inputs and will be able to populate the arrayselection before hitting apply.

is there a way for the users to provide a string as a separate parameter (outside of the SetFileName) method before hitting apply?

Not with python plugins, no.

Hi Mathieu,
Could you direct me to where I can learn more on how to create a separate string parameter (outside of the SetFileName method) before hitting apply? Are there examples of doing something similar to this via the C++ API? Or will this require modification to the C++ source code?
Thank you,
Victor

Are there examples of doing something similar to this via the C++ API?

It is doable but I would not recommend it.
The problem is that it is in contradiction with the way paraview works.

What is the problem with the solution with a reader and a filter I’m suggesting ?

Hey Matthieu,
I think the separation of reader and filter should work. However, I was hoping for a solution where I can apply a single filter. I was thinking on the lines of how Paraview currently loads in a file, where the user is able to select which datasets in the file to load before hitting apply.

Is the current Paraview loader also implemented with separate reader and filter as well?
Thanks,
Victor

how Paraview currently loads in a file

This just a single reader. But readers have no input.

In your case you have a filter with an input AND a filename, that’s why you have to separate.

Hi Matthieu,
I am now trying to implement the reader and filter separately. After creating the reader using the @smproxy.reader I went to the tools>Manage Plugins… tab in Paraview and loaded the plugin with no issues. However, when I go to File>Open… tab in Paraview, I do not see my custom reader in the ‘‘Files of type:’’ drop down list.

If I switch @smproxy.reader to @smproxy.source, and load the plugin the same way, my custom plugin does show up. Is there something extra I need to do to expose the reader in plugin?

Furthermore, the data names which I want to load into the array selection list are not known until I am able to specify which file I intend to load in. However, the specified filename that I select through the GUI that is provided by the @smproperty.dataarrayselection does not get registered via SetFileName until after hitting apply. Is there a way to register the filename before hitting apply? I can only populate the array selection list before hitting apply if I have a handle on the filename.
Thanks,
Victor

Here is how it should look :

@smproxy.reader(name="PythonCSVReader", label="Python-based CSV Reader",
                extensions="csv", file_description="CSV files")

Hi Matthieu,
I am finding that the @smproxy.reader decorator has an issue with the h5 extension. I have pared down my reader to simply set the filename to illustrate the issue.

When I try the following, the reader works fine:

@smproxy.reader(name="MyReader", label="my custom reader", 
                extensions="csv", file_description="CUSTOM files")
class MyReader(VTKPythonAlgorithmBase):
    def __init__(self):
        VTKPythonAlgorithmBase.__init__(self,
                                        nInputPorts=0, 
                                        nOutputPorts=1, 
                                        outputType="vtkTable")
        self._filename = None

    @smproperty.stringvector(name="File Name")
    @smdomain.filelist()
    @smhint.filechooser(extensions="csv", file_description="CUSTOM files")
    def SetFileName(self, name):
        if self._filename != name:
            self._filename = name
            self.Modified()

I’ve also tried setting the extensions keyword to other file extensions such as “txt” and “inp” and the reader also worked fine. However when the extensions keyword is set to “h5”, Paraview crashes with a segmentation fault. The code where I switch the extension to “h5” is shown below:

@smproxy.reader(name="MyReader", label="my custom reader", 
                extensions="h5", file_description="CUSTOM files")
class MyReader(VTKPythonAlgorithmBase):
    def __init__(self):
        VTKPythonAlgorithmBase.__init__(self,
                                        nInputPorts=0, 
                                        nOutputPorts=1, 
                                        outputType="vtkTable")
        self._filename = None

    @smproperty.stringvector(name="File Name")
    @smdomain.filelist()
    @smhint.filechooser(extensions="h5", file_description="CUSTOM files")
    def SetFileName(self, name):
        if self._filename != name:
            self._filename = name
            self.Modified()

Note that the only thing that changed is the extension that is passed to the “extensions” keyword.
Thanks,
Victor

Your script misses from paraview.util.vtkAlgorithm import * at the start.
Other than that, this works fine on my side.

Which version of ParaView are you using ?

Hi Mathieu,
It turns out the issue with the h5 extension is due to an inconsistency in the versions of hdf5 installed with paraview and h5py.

Is there an example of how to write my RequestData for parallel processing in the client-server mode? Currently, the code written above only works properly in serial.

Thank you,
Victor

I’m not sure we have a python example for parallel reading yet, but we should !

You can take a look into C++ example in VTK sources for insipiration.

What about examples for parallel appending of data? In this case, it would be a parallel filter rather than a parallel reader right?

Could you point me to the specific C++ examples that you are referring to?
Thank you,
Victor

Parallel filter :
vtkPStreamTracer.cxx

Parallel Reader :
vtkPDataSetReader.cxx

1 Like