append an array as a new vector/scalar field from pandas dataframe with programmable filter and paraview's python console

ParaView Support
1

I have a dataframe that is of the following format

ID  t_0       t_1     t_2     t_n
0    .         .       .       .    
.
.
m

and I want to read and append each column array as a field at their respective time-step to an existing set of vtu files (vtkXMLUnstructuredGrid) so that I can build further filters in paraview and create a pipeline.

Part1 - Using the Programmable Filter

frame = pd.read_hdf('/path/to/file/frame.h5', 'table', mode='r')

ts = self.GetInputDataObject(0,0).GetInformation().Get(vtk.vtkDataObject.DATA_TIME_STEP())

t = int(ts)

wantedArray = frame.filter(['t_{}'.format(t)], axis=1)

from numpy import array

a = array(wantedArray)

output.PointData.append(a, "newField")

The above script works. However, as the dataframe is quite large (~10 GB) it makes the whole process slow when I move to the next time step because the dataframe is read again while updating the pipeline. Is there a way to read the large .h5 file once and store it in the memory and access the data?

Part 2 - Using paraview's python console

>>> frame = pd.read_hdf('/path/to/file/frame.h5', 'table', mode='r')
>>> view = GetActiveView()
>>> t = int(view.ViewTime)
>>> inputSource = GetActiveSource()
>>> inputData = servermanager.Fetch(inputSource)
>>> nPoints=inputData.GetNumberOfPoints()
>>> newField=paraview.vtk.vtkFloatArray()
>>> newwField.SetNumberOfValues(nPoints)
>>> newField.SetName("newField")

>>> wantedArray = frame.filter(['t_{}'.format(t)], axis=1)

>>> from numpy import array
>>> a = array(wantedArray)

>>> for k in range(nPoints):
    ...newField.SetValue(k, a[k])

>>> inputData.GetPointData().AddArray(newField)

The above code creates the correct field and I am able to access the values in the python console with

inputData.GetPointData().GetArray("newField").GetValue()

The problem is that I am not able to visualize this array in the render view and process it further with other filters. How do I get this array back to show up in the render view (gui) as a field in the drop down list?

I tried few things like Show(), Render() but did not work. I also tried the follwing snippet:

t = TrivialProducer() 
filter = t.GetClientSideObject()
filter.SetOutput(inputData)
t.UpdatePipeline()

but this throws the following error:

AttributeError: 'NoneType' object has no attribute 'SetOutput'

Any alternate suggestions are also welcome. I want to implement this as efficiently as possible!

2

I’d stick with the Progammable Source approach than the Pyhton Console way. A mildly clunky way of saving state/cache is to use custom attributes on self. e.g.

if not hasattr(self, 'frame'):
   frame = pd.read_hdf(...)
   setattr(self, 'frame', frame)
else:
  frame = self.frame

In ParaView 5.6, Python Algorithm makes it possible to add Python classes for readers/filters/writers etc. There, you can easily implement caching mechanisms as needed using class members.