Issue with Programmable Filter on OSMesa compiled version for Unix, but not with Windows installation.

The problem
I use a script to convert PointData to CellData in the programmable filter (the netCDFreader returns PointData). Cory once wrote it and it works fine on my Windows Laptop with the normal Paraview Installation (v5.4.1). However, on my OSMesa compiled version on Linux (Paraview 5.4.1, Mesa 17.2.8), it seizes to work. For some reason my Programmable Filter does not seem to do anything, even though no error is thrown, but no data is passed. When I remove the Programmable Filter, to just plot PointData, everything renders fine again.

I’m interested in your thoughts as to the cause of this problem.

The things I tried
I first thought it was a python path error only in the programmable filter. But it does not throw any import errors and when I print my sys.path in the filter script, it is identical to sys.path in my pvbatch script, which has no trouble with processing data.

Furthermore, when I define the following function to print the variable names in a dataset (and to distuingish cell data or point data).

 def get_arraynames(Property):
     names = {}
     
     cd = servermanager.Fetch(Property).GetCellData()
     names["cd"] = [cd.GetArrayName(i) for i in range(cd.GetNumberOfArrays())]
     
     pd = servermanager.Fetch(Property).GetPointData()
     names["pd"] = [pd.GetArrayName(i) for i in range(pd.GetNumberOfArrays())]
     return(names)

All Properties in the pipeline after my programmable filter on the Windows version get a list under “cd”, however this list is empty on the Linux version I compiled. (“pd” is also empty for the filter in that version. Before the programmable filter, “pd” contains the names both on Linux as Windows, as expected)

The filter script is hardcoded in my pvbatch script as a 1 line string with “\n” (as traced by Paraview on Windows), so I think it should not be a carriage return Windows-Linux conversion thing.

The filter script

dims = inputs[0].GetDimensions()
ext = inputs[0].GetExtent()
output.SetDimensions(dims[0]+1, dims[1]+1, dims[2]+1)
output.SetExtent(ext[0], ext[1]+1, ext[2], ext[3]+1, ext[4], ext[5]+1)
inputPd = inputs[0].PointData
outputCd = output.CellData
for array in inputPd:
   print(type(array))
   outputCd.append(array, array.GetName())

bash script with which I compiled Paraview:

 #!/bin/bash

module load cmake

mkdir paraview.bld
cd paraview.bld

MESA_INSTALL_PREFIX=$HOME/visualization/mesa
LLVM_INSTALL_PREFIX=$HOME/visualization/llvm

export PATH=$PATH:$HOME/visualization/llvm/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${MESA_INSTALL_PREFIX}/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${LLVM_INSTALL_PREFIX}/lib

cmake \
  -DPARAVIEW_ENABLE_PYTHON=ON                                  \
  -DCMAKE_INSTALL_PREFIX=$HOME/visualization/paraview \
  -DPYTHON_EXECUTABLE=$HOME/anaconda3/envs/py27/bin/python2.7  \
  -DPYTHON_INCLUDE_DIR=$HOME/anaconda3/envs/py27/include/python2.7  \
  -DPYTHON_LIBRARY=$HOME/anaconda3/envs/py27/lib/libpython2.7.so.1.0 \
  -DPARAVIEW_BUILD_QT_GUI=OFF                                  \
  -DVTK_USE_X=OFF                                              \
  -DOPENGL_INCLUDE_DIR=IGNORE                                  \
  -DOPENGL_xmesa_INCLUDE_DIR=IGNORE                            \
  -DOPENGL_gl_LIBRARY=IGNORE                                   \
  -DOSMESA_INCLUDE_DIR=${MESA_INSTALL_PREFIX}/include          \
  -DOSMESA_LIBRARY=${MESA_INSTALL_PREFIX}/lib/libOSMesa.so \
  -DVTK_OPENGL_HAS_OSMESA=ON                                   \
  -DVTK_USE_OFFSCREEN=OFF                                      \
  ../ParaView-v5.4.1
 
make -j32
make -j32 install

Are there any suggestions as to what I should look for to debug this problem further? Or how I can improve this problem description to increase the ease of responding (in other words: how I can help you in helping me)?

Since this is image data, is your programmable filter also providing the RequestInformation script? You will need to provide that too to correctly report the data extents being generated since they are different from the input.

No, I have not, thanks. The windows version strangely enough did not require this. Are there any example scripts available somewhere?

Yes, please see this example: https://www.paraview.org/Wiki/Python_Programmable_Filter#Changing_Data_Type

Thank you, I have added the following script to RequestInformation Script:

from paraview import util

ext = inputs[0].GetExtent()

util.SetOutputWholeExtent(self, [ext[0], ext[1], ext[2], ext[3], ext[4], ext[5]])

Again it works fine on Windows (interestingly enough I had to remove the +1 behind ext[1],ext[3], and ext[5]), but I’m getting the same (Python) errors I was getting before on Unix:

Traceback (most recent call last):
  File "<string>", line 22, in <module>
  File "<string>", line 4, in RequestInformation
TypeError: 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "<string>", line 22, in <module>
  File "<string>", line 5, in RequestData
TypeError: 'NoneType' object has no attribute '__getitem__'

I also see there is an RequestUpdateExtent script, should I add something there as well?

Furthermore, from the above error description it seems that the filter already gets a None before it even can do something with the RequestInformation scripts, despite that point data is going into the ProgrammableFilter.

Summary of printouts in this pseudo-code:

print(get_arraynames(netCDFReader1))
#Prints
#{'pd': ['conc', 'dcdt', 'fhead', 'head', 'vx', 'vy', 'vz'], 'cd': []}
programmableFilter1 = ProgrammableFilter(Input=netCDFReader1)
#...
#Here all the scripts etc. are assigned to programmableFilter1
#...
print(get_arraynames(programmableFilter1))
#Prints
#{'pd': [], 'cd': []}

My programmable filter script now depends on inputs which is None in the Unix version I run and a vtk.numpy_interface.dataset_adapter.DataSet object in the Windows version.

Update: So apparently it sets hasnumpy to False on the Unix version.

You build the linux version yourself, correct? If you built against an installed version of Python, you’ll need to install numpy with either your package manager or pip so that it is available in your Python installation.

Yes correct,

The Anaconda environment I am using does have numpy installed

I think I’m getting Python environments mixed as I include a anaconda environment but the installed build also has its own python environment.

Should I use -DPARAVIEW_ENABLE_NUMPY=ON when building?

Update: Nope that did not fix it.

Backing up a bit, what are you using to build ParaView? Are you using the superbuild?

ParaView’s Python environment is taken from whichever one you build against. Are you building against the Anaconda environment?

That’s not a defined option in ParaView or ParaView’s superbuild AFAIK.

I am building ParaView from a ParaView-v5.4.1.tar.gz file I found on a portal (forgot the link, since I downloaded it a month ago).
I followed these instructions https://www.paraview.org/Wiki/ParaView/ParaView_And_Mesa_3D

I am not sure what “building against” means here, I installed a conda python 2.7 environment with numpy and I used these options:

-DPYTHON_EXECUTABLE=$HOME/anaconda3/envs/py27/bin/python2.7
-DPYTHON_INCLUDE_DIR=$HOME/anaconda3/envs/py27/include/python2.7
-DPYTHON_LIBRARY=$HOME/anaconda3/envs/py27/lib/libpython2.7.so.1.0 \

Before building, I ensured that this environment is added to $PYTHONPATH.

However, if I add this environment to $PYTHONPATH before running pvbatch, I get segmentation faults.

Maybe a dumb question, but if you run $HOME/anaconda3/envs/py27/bin/python2.7, does import numpy work?

Good that you also force me to check the basics (as this is probably a silly mistake I made), but unfortunately it is not this one:

$ $HOME/anaconda3/envs/py27/bin/python2.7
Python 2.7.15 | packaged by conda-forge | (default, Feb 28 2019, 04:00:11)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.zeros((10))
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

OK, I got a bit further with bug tracking. When I do print(sys.path) in the Filter script, it outputs:
['/nfs/home1/jengelen/synthdelta/delta_aquifer/post/paraview_macros', '/home/jengelen/visualization/paraview/lib/paraview-5.4/site-packages/vtk', '/home/jengelen/visualization/paraview/lib/paraview-5.4/site-packages', '/home/jengelen/visualization/paraview/lib/paraview-5.4', '/nfs/home1/jengelen/synthdelta/jobs', '/home/jengelen/visualization/paraview/lib/paraview-5.4/site-packages', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

So apparently it built against a python2.7 installation in the /usr/ folder on the cluster instead of the folders I specified (I guess the Red Hat Enterprise server comes with this by default).

OK, I tried rebuilding, ensuring that the anaconda environment was activated and in my PYTHONPATH, but it did not work. Even though during building it prints:
-- Found PythonInterp: /home/jengelen/anaconda3/envs/py27/bin/python2.7 (found suitable version "2.7.15", minimum required is "2")

The /usr/lib/python2.7 keeps on popping up in the filter script.

It is probably easier to make the filter script independent of numpy. So by not using inputs, but use self.GetIntput() or something. Is that possible?

OK, I have fixed my problem by getting rid of the numpy dependency, as it was unnecessary.

To avoid having to use inputs, I changed my programmable filter script to:

inp = self.GetInput()
dims = inp.GetDimensions()
ext = inp.GetExtent()

oup = self.GetOutput()
oup.SetDimensions(dims[0]+1, dims[1]+1, dims[2]+1)
oup.SetExtent(ext[0], ext[1]+1, ext[2], ext[3]+1, ext[4], ext[5]+1)

N=inp.GetPointData().GetNumberOfArrays()

for i in range(N):
    data = inp.GetPointData().GetAbstractArray(i)
    oup.GetCellData().AddArray(data)

Thanks Cory and Utkarsh for the support, your questions guided me in the right direction!

Yep, that will work if you don’t mind giving up the numpy support.

@ben.boeckel Any ideas for getting ParaView to build with Joeri’s Anaconda installation instead of the Python in /usr/lib?

I’d just go through the CMakeCache.txt and find any mention of Python from /usr/lib and change it to the one in the Anaconda env.