# append an array as a new vector/scalar field from pandas dataframe with programmable filter and paraview's python console

**URL:** https://discourse.paraview.org/t/append-an-array-as-a-new-vector-scalar-field-from-pandas-dataframe-with-programmable-filter-and-paraviews-python-console/482
**Category:** ParaView Support
**Tags:** python
**Created:** [August 27, 2018, 8:58pm UTC](https://discourse.paraview.org/t/append-an-array-as-a-new-vector-scalar-field-from-pandas-dataframe-with-programmable-filter-and-paraviews-python-console/482 "2018-08-27T20:58:10Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![Thangam](https://discourse.paraview.org/letter_avatar_proxy/v4/letter/t/e95f7d/32.png) [@Thangam](https://discourse.paraview.org/u/Thangam)
#### Post date: [August 27, 2018, 8:58pm UTC](https://discourse.paraview.org/t/append-an-array-as-a-new-vector-scalar-field-from-pandas-dataframe-with-programmable-filter-and-paraviews-python-console/482/1 "2018-08-27T20:58:10Z")

</div>

I have a dataframe that is of the following format

```
ID t_0 t_1 t_2 t_n
0 . . . .    
.
.
m

```

and I want to read and append each column array as a field at their respective time-step to an existing set of vtu files (`vtkXMLUnstructuredGrid`) so that I can build further filters in paraview and create a pipeline.

`Part1 - Using the Programmable Filter`

```
frame = pd.read_hdf('/path/to/file/frame.h5', 'table', mode='r')

ts = self.GetInputDataObject(0,0).GetInformation().Get(vtk.vtkDataObject.DATA_TIME_STEP())

t = int(ts)

wantedArray = frame.filter(['t_{}'.format(t)], axis=1)

from numpy import array

a = array(wantedArray)

output.PointData.append(a, "newField")

```

The above script works. However, as the dataframe is quite large (~10 GB) it makes the whole process slow when I move to the next time step because the dataframe is read again while updating the pipeline. Is there a way to read the large .h5 file once and store it in the memory and access the data?

`Part 2 - Using paraview's python console`

```
>>> frame = pd.read_hdf('/path/to/file/frame.h5', 'table', mode='r')
>>> view = GetActiveView()
>>> t = int(view.ViewTime)
>>> inputSource = GetActiveSource()
>>> inputData = servermanager.Fetch(inputSource)
>>> nPoints=inputData.GetNumberOfPoints()
>>> newField=paraview.vtk.vtkFloatArray()
>>> newwField.SetNumberOfValues(nPoints)
>>> newField.SetName("newField")

>>> wantedArray = frame.filter(['t_{}'.format(t)], axis=1)

>>> from numpy import array
>>> a = array(wantedArray)

>>> for k in range(nPoints):
    ...newField.SetValue(k, a[k])

>>> inputData.GetPointData().AddArray(newField)

```

The above code creates the correct field and I am able to access the values in the python console with

```
inputData.GetPointData().GetArray("newField").GetValue()

```

The problem is that I am not able to visualize this array in the render view and process it further with other filters. How do I get this array back to show up in the render view (gui) as a field in the drop down list?

I tried few things like `Show()`, `Render()` but did not work. I also tried the follwing snippet:

```
t = TrivialProducer() 
filter = t.GetClientSideObject()
filter.SetOutput(inputData)
t.UpdatePipeline()  

```

but this throws the following error:

`AttributeError: 'NoneType' object has no attribute 'SetOutput'`

Any alternate suggestions are also welcome. I want to implement this as efficiently as possible!

---

<div class="post-metadata">

### Author: ![utkarsh.ayachit](https://discourse.paraview.org/user_avatar/discourse.paraview.org/utkarsh.ayachit/32/39_2.png) [@utkarsh.ayachit](https://discourse.paraview.org/u/utkarsh.ayachit)
#### Post date: [August 28, 2018, 12:02am UTC](https://discourse.paraview.org/t/append-an-array-as-a-new-vector-scalar-field-from-pandas-dataframe-with-programmable-filter-and-paraviews-python-console/482/2 "2018-08-28T00:02:33Z")

</div>

I’d stick with the **Progammable Source** approach than the **Pyhton Console** way. A mildly clunky way of saving state/cache is to use custom attributes on `self`. e.g.

```python
if not hasattr(self, 'frame'):
   frame = pd.read_hdf(...)
   setattr(self, 'frame', frame)
else:
  frame = self.frame

```

In ParaView 5.6, [Python Algorithm](https://www.paraview.org/ParaView/Doc/Nightly/www/py-doc/paraview.util.vtkAlgorithm.html) makes it possible to add Python classes for readers/filters/writers etc. There, you can easily implement caching mechanisms as needed using class members.