Plugin Reader Based on Another Plugin

Development
1

Hello. I have built a Python-based reader plugin that works just fine for a single file.
It’s actually the reader I asked for help with here

But now I would like to use it to open a series of files. I’ve been attempting to use a simple XML-based plugin to create a vtkFileSeriesReader that references my single-file reader. This second plugin loads fine, but when used to open a file (or file series) it crashes because the SetFileName method of the underlying reader is never called.

The FileSeriesReader XML:

<ServerManagerConfiguration>
  <ProxyGroup name="sources">
    <SourceProxy class="vtkFileSeriesReader"
                 file_name_method="SetFileName"
                 label="My XDMF Reader Series"
                 name="MyXDMFReaderSeries"
                 si_class="vtkSIMetaReaderProxy">
      <Documentation long_help="Read Lagrangian XDMF data set from from a xmf dataset."
                     short_help="Read Lagrangian XDMF data set from from a xmf dataset.">
                     This XDMF reader loads data stored in xmf format. The 
                     output of this reader is a XDMF Dataset
                     dataset.</Documentation>
      <SubProxy>
        <!-- proxygroup="internal_sources" -->
        <Proxy name="Reader"
               proxygroup="sources"
               proxyname="MyXDMFReader"></Proxy>
        <ExposedProperties>
        </ExposedProperties>
      </SubProxy>
      <StringVectorProperty animateable="0"
                            clean_command="RemoveAllFileNames"
                            command="AddFileName"
                            name="FileName"
                            number_of_elements="0"
                            panel_visibility="default"
                            repeat_command="1">
        <FileListDomain name="files" />
        <Documentation>The list of files to be read by the 
        reader.</Documentation>
      </StringVectorProperty>
      <DoubleVectorProperty information_only="1"
                            name="TimestepValues"
                            repeatable="1">
        <TimeStepsInformationHelper />
        <Documentation>Available timestep values.</Documentation>
      </DoubleVectorProperty>
      <Hints>
        <ReaderFactory extensions="xmf"
                       file_description="Lagrangian Files" />
      </Hints>
    </SourceProxy>
  </ProxyGroup>
</ServerManagerConfiguration>

Perhaps the first question should be: Is it even possible to create a plugin that references another plugin? The above XML seems to pick up the previously loaded plugin just fine, as I can see references to it in the error, as seen below:

ERROR: In /home/shelf1/compile/visualization/src/paraview/5.6.0_gui/VTK/IO/Xdmf2/vtkXdmfReader.cxx, line 200
vtkXdmfReader (0x558a02db6ae0): Error opening file None

ERROR: In /home/shelf1/compile/visualization/src/paraview/5.6.0_gui/VTK/Common/ExecutionModel/vtkExecutive.cxx, line 782
vtkCompositeDataPipeline (0x558a02db5a00): Algorithm vtkXdmfReader(0x558a02db6ae0) returned failure for request: vtkInformation (0x558a02c709f0)
  Debug: Off
  Modified Time: 849987
  Reference Count: 1
  Registered Events: (none)
  Request: REQUEST_DATA_OBJECT
  ALGORITHM_AFTER_FORWARD: 1
  FORWARD_DIRECTION: 0

So if there is no issue with my general approach, what am I missing from this XML plugin?

Thanks,
Ted

2

Perhaps an alternative approach is to modify my base reader to itself use the vtkFileSeriesReader class?

3

You should use the FileSeries only in the xml side, like you have done in the code you shown.
Hard to say why it doesn’t work without testing it, you may want to take a look into the already working reader that uses the FileSeries in ParaView.

the XMLPolyDataReader come to mind.

4

I have run into the same issue using a simple python based reader plugin based on the csv reader example.

With some debug print statements to the SetFileName method of the reader I can see it actually gets called. It just get passed None for some reason.

Is there something that needs to be added to the underlying reader plugin beyond the very basic RequestData functionality to make the wrapping work? My assumption was that the Wrapper takes care of all the time relevant parts and just supplies an appropriate filename for the wrapped source.

My python plugin works when used for single files without the wrapper plugin from the xml file. The xml plugin file series works when i switch in an existing SourceProxy like the vtkPDataSetReader instead of my plugin.

5

Which same issue ? Please include code and error messages.

6

This is the python based reader i use for testing cobbled together from some of the examples

readertest.py

from paraview.util.vtkAlgorithm import *
from vtk.numpy_interface import algorithms as algs
from vtk.numpy_interface import dataset_adapter as dsa
import numpy as np
import vtk

@smproxy.reader(name="PythonCSVReader", label="Python-based CSV Reader",
extensions="csv", file_description="CSV files")
class PythonCSVReader(VTKPythonAlgorithmBase):
    def __init__(self):
        VTKPythonAlgorithmBase.__init__(self, nInputPorts=0, nOutputPorts=1, outputType='vtkPolyData')
        self._filename = None


    @smproperty.stringvector(name="FileName")
    @smdomain.filelist()
    @smhint.filechooser(extensions="csv", file_description=" CSV files")
    def SetFileName(self, name):
        """Specify filename for the file to read."""
        print("setname: ",name)
        if self._filename != name:
            self._filename = name
            self.Modified()

    def RequestData(self, request, inInfoVec, outInfoVec):
        from vtkmodules.vtkCommonDataModel import vtkPolyData
        output = vtkPolyData.GetData(outInfoVec, 0)
        tt = self._filename
        data = np.genfromtxt(tt,dtype=None, names=True,delimiter=',',autostrip=True)
        # convert the 3 arrays into a single 3 component array for
        # use as the coordinates for the points.
        coordinates = algs.make_vector(data["X"],data["Y"],data["Z"])
        # create a vtkPoints container to store all the
        # point coordinates.
        pts = vtk.vtkPoints()
        # numpyTovtkDataArray is needed to called directly to convert the NumPy
        # to a vtkDataArray which vtkPoints::SetData() expects.
        pts.SetData(dsa.numpyTovtkDataArray(coordinates , "Points"))
        # set the pts on the output.
        output.SetPoints(pts)
        # next, we define the cells i.e. the connectivity for this mesh.
        # here, we are creating merely a point could, so we’ll add
        # that as a single poly vextex cell.
        numPts = pts.GetNumberOfPoints()
        # ptIds is the list of point ids in this cell
        # (which is all the points)
        ptIds = vtk.vtkIdList()
        ptIds.SetNumberOfIds(numPts )
        for a in range( numPts ):
            ptIds.SetId(a , a)
            # Allocate space for 1 cell.
            output.Allocate (1)
            output.InsertNextCell( vtk.VTK_POLY_VERTEX , ptIds )
        return 1

It reads x,y,z data from a CSV file as single points. An example csv file could be
data_1.csv:

X,Y,Z
1,2,3
4,5,2
8,5,1
4,1,6
3,2,2

This works well in paraview.
My goal is now to wrap this in a FileSeries reader to make use of the automatic time series treatment of numbered input files just as the OP.

As first start i have created a slightly modified version one of the existing reader definitions to make sure this works as i expect. (I changed the extension from vtk to vtka to make sure its actually the new reader being called.)

<ServerManagerConfiguration>
 <ProxyGroup name="internal_sources">
   <SourceProxy name="readertest" 
                class="vtkPDataSetReader"
                label="Legacy VTK reader aaa">
     <StringVectorProperty
        name="FileName"
        animateable="0"
        command="SetFileName"
        number_of_elements="1">
        <FileListDomain name="files"/>
        <Documentation>
          This property specifies the file name for the Legacy VTK reader.
        </Documentation>
     </StringVectorProperty>
     <Hints>
      <ReaderFactory extensions="vtka"
          file_description="Legacy VTK files" />
     </Hints>
   </SourceProxy>
  </ProxyGroup>

  <ProxyGroup name="sources">
   <SourceProxy name="seriesreadertest"
                          class="vtkFileSeriesReader"
                          si_class="vtkSIMetaReaderProxy"
                          label="Legacy VTK reader aaa"
                          file_name_method="SetFileName">
     <SubProxy>
        <Proxy name="Reader"
          proxygroup="internal_sources" proxyname="readertest">
        </Proxy>
     </SubProxy>

      <StringVectorProperty name="FileNameInfo"
        command="GetCurrentFileName"
        information_only="1" >
        <SimpleStringInformationHelper />
     </StringVectorProperty>

     <StringVectorProperty
        name="FileNames"
        clean_command="RemoveAllFileNames"
        command="AddFileName"
        animateable="0"
        number_of_elements="0" 
        repeat_command="1">
        <FileListDomain name="files"/>
     </StringVectorProperty>

     <DoubleVectorProperty 
        name="TimestepValues"
        repeatable="1"
        information_only="1">
        <TimeStepsInformationHelper/>
     </DoubleVectorProperty>

     <Hints>
      <ReaderFactory extensions="vtka"
          file_description="Legacy VTK files aaa" />
     </Hints>
   </SourceProxy>
</ProxyGroup>
</ServerManagerConfiguration>

Saving two cone sources as legacy vtk and naming them cone_1.vtka and cone_2.vtka results in the desired effect.

The problem starts if i try to wrap my python based csv reader from above. I use the following definition:

<ServerManagerConfiguration>
  <ProxyGroup name="sources">
   <SourceProxy name="TimeTestFileReader"
                          class="vtkFileSeriesReader"
                          si_class="vtkSIMetaReaderProxy"
                          label="time test reader csv"
                          file_name_method="SetFileName">
     <SubProxy>
        <Proxy name="Reader"
          proxygroup="sources" proxyname="PythonCSVReader">
        </Proxy>
     </SubProxy>

      <StringVectorProperty name="FileNameInfo"
        command="GetCurrentFileName"
        information_only="1" >
        <SimpleStringInformationHelper />
     </StringVectorProperty>

     <StringVectorProperty
        name="FileNames"
        clean_command="RemoveAllFileNames"
        command="AddFileName"
        animateable="0"
        number_of_elements="0"
        repeat_command="1">
        <FileListDomain name="files"/>
     </StringVectorProperty>

     <DoubleVectorProperty
        name="TimestepValues"
        repeatable="1"
        information_only="1">
        <TimeStepsInformationHelper/>
     </DoubleVectorProperty>

     <Hints>
      <ReaderFactory extensions="csv"
          file_description="csv timetest" />
     </Hints>
   </SourceProxy>
</ProxyGroup>
</ServerManagerConfiguration>

Using this definition i get the following error message indicating a filename of none is passed to numpy:

(  20.960s) [paraview        ] vtkPythonAlgorithm.cxx:112    ERR| vtkPythonAlgorithm (0x55fbe960c050): Failure when calling method: "ProcessRequest":
Traceback (most recent call last):
  File "/mnt/hdd/build/paraview/src/build/lib/python3.7/site-packages/vtkmodules/util/vtkAlgorithm.py", line 152, in ProcessRequest
    return vtkself.ProcessRequest(request, inInfo, outInfo)
  File "/mnt/hdd/build/paraview/src/build/lib/python3.7/site-packages/vtkmodules/util/vtkAlgorithm.py", line 198, in ProcessRequest
    return self.RequestData(request, inInfo, outInfo)
  File "paraplugintest/singlereader.py", line 36, in RequestData
    data = np.genfromtxt(tt,dtype=None, names=True,delimiter=',',autostrip=True)
  File "/usr/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1759, in genfromtxt
    fid = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/usr/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/usr/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
    raise IOError("%s not found." % path)
OSError: None not found.

In the debug output i can see that the SetFileName method of my python reader containing the

print("setname: ",name)

called by the wrapper, it just gets passed a name of None resulting in the above error.

setname:  None

This looks to me as the same issue the OP had and as something to do with the interaction of the wrapper and the reader.

Is there anything wrong with the way i have set this up? Maybe I’m missing something?

7

The problem starts if i try to wrap my python based csv reader from above.

I’m not sure this is supported. @utkarsh.ayachit may know more.

8

This issue came back to the top of the stack so I went and bugged Utkarsh :grinning:

The definite answer: wrapping a python plugin with xml is not supported without significant work.

The recommendation: handle multi-file support directly in the python reader. From Utkarsh:

It is simply a matter of adding a AddFileName,
RemoveAllFileNames API and defining an appropriate XML for the same.

e.g.

@smproperty.xml("""
  <StringVectorProperty animateable="0"
                            clean_command="RemoveAllFileNames"
                            command="AddFileName"
                            name="FileName"
                            number_of_elements="1"
                            panel_visibility="never"
                            repeat_command="1">
        <FileListDomain name="files" />
      </StringVectorProperty>
"""
def AddFileName(self, name):
    ...

def RemoveAllFileNames():
   ...

The stuff about reporting timesteps available, etc. is same as the
PythonCSVReader in PythonAlgorithmExamples.py

9

Wrapping a Python Algorithm based reader as the internal reader to be used in a FileSeries meta-reader is what’s not supported. The issue is this. That code explicitly calls SetFileName using CS wrapping which won’t be possible on the PythonAlgorithm.

Another workaround could be be writer a new subclass of vtkFileSeriesReader that forwards this invocation to PythonAlgorithm using appropriate Python-C API.