Superbuild with Anaconda Python 3.7.1

TimJim · March 5, 2019, 4:41am

Hi, I’m attempting to build the Paraview 5.6 superbuild with Python 3 so that I can use PVPython with my existing code. I seem to be getting an issue with ‘undefined references’ similar to:

…/lib/libvtkWrappingPython37Core-pv5.6.so.1: undefined reference to `PyUnicode_InternFromString’

…/lib/libvtkPVServerManagerCore-pv5.6.so.1: undefined reference to `PyLong_FromLong’

and more of the same. I’ve tried building with Python 3.6 and 3.7 with similar results. To troubleshoot, I turned off all settings and only changed the following variables in my CMakeCache.txt:
CMAKE_INSTALL_PREFIX: /opt/paraview/ParaView-5.6.0
PYTHON_EXECUTABLE: /home/tjim/anaconda3/envs/pv/bin/python
PYTHON_INCLUDE_DIR: /home/tjim/anaconda3/envs/pv/include/python3.7m
PYTHON_LIBRARY: /home/tjim/anaconda3/envs/pv/lib/python3.7
PYTHON_LIBRARY_DEBUG: *pdb.py
python_FIND_LIBRARIES: *ON
superbuild_python_pip: /home/tjim/anaconda3/envs/pv/bin/pip

Can someone give me some pointers on how to build with Python 3?
Thanks and regards,
Tim

mwestphal · March 5, 2019, 9:28am

Looks like you are trying to build with your own Anaconda Python install instead of building Python within the superbuild, is that correct ?

TimJim · March 5, 2019, 9:50am

Hi, yes that’s right!

mwestphal · March 5, 2019, 9:51am

Is the superbuild successful when building its own python ?

TimJim · March 5, 2019, 10:01am

It appears to build successfully with it’s own python 2.7.

ben.boeckel · March 5, 2019, 3:57pm

I fear the superbuild might be a bit Python2-assumption heavy at the moment. Is there a specific reason you’re using the superbuild rather than trying to build a standalone ParaView source tree against Anaconda?

TimJim · March 5, 2019, 4:22pm

Only that I originally thought it might save me the trouble of locating and compiling all the other libraries required! Is there a minimal version that I could compile? I mainly would like to visualise flow blocks and perform simple slicing. Many thanks!

ben.boeckel · March 5, 2019, 4:40pm

ParaView ships most of its third party dependencies with its own source code. I think the only things that the superbuild provides on top are Python (which you have already), and these libraries (most of which are, I assume, easily obtainable from Anaconda if necessary):

Mesa
ADIOS (I/O file format)
ParaViewWeb dependencies (should all be available via pip)
libLAS (I/O file format)
Boost (used in some libraries, but nothing “core”)
FFmpeg (also not critical)
Fontconfig/freetype
Qt5
an MPI

The other libraries in the superbuild are there so that ParaView can share with these dependencies (e.g., zlib).

TimJim · March 6, 2019, 11:36am

Thanks for the email. I’ll give the standard build a go!

Are the items on the list all dependencies? I had a search for ParaViewWeb on pip but was only able to find wslink - is it sufficient to install that?

Thanks and regards.

mwestphal · March 6, 2019, 11:51am

Qt (if you need the GUI) and FontConfig are the only required dependencies.

TimJim · March 6, 2019, 11:56am

Thank you! I am currently trying to generate using ccmake and I’m getting a lot of warnings along the lines of:

WARNING: Target "vtkPVPythonCatalystPythonD" requests linking to directory "/home/tjim/anaconda3/envs/pv/lib/python3.7".  Targets may link

only to libraries.  CMake is dropping the item.

WARNING: Target "vtkPVPythonCatalystPython" requests linking to directory "/home/tjim/anaconda3/envs/pv/lib/python3.7".  Targets may link

only to libraries.  CMake is dropping the item.

WARNING: Target "vtkPythonInterpreter" requests linking to directory "/home/tjim/anaconda3/envs/pv/lib/python3.7".  Targets may link only

to libraries.  CMake is dropping the item.

Are these safe to ignore?

Kind regards,

Tim

mwestphal · March 6, 2019, 11:58am

They are safe to ignore.

TimJim · March 6, 2019, 3:38pm

Thanks, Mathieu. I tried to make the build but it seems that I’m getting another issue with mpi4py. It seems to try to grab the version in my python directory and also in the Paraview directories. Can you give me some tips on how to fix this? Thank you. Here are are a few lines of the errors:

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/message_lite.cc.o

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/init.py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/main.py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include/mpi4py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/lib-pmpi’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/init.py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/main.py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include/mpi4py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/lib-pmpi’…

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/ArrayRangeCompute.cxx.o

[ 3%] Built target mpi4py

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/CellSetExplicit.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/repeated_field.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlsave.c.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pq3DViewEventPlayer.cxx.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pq3DViewEventTranslator.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/wire_format_lite.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlschemas.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlschemastypes.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlstring.c.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/coded_stream.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/zero_copy_stream.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/zero_copy_stream_impl_lite.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlunicode.c.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractActivateEventPlayer.cxx.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractBooleanEventPlayer.cxx.o

Scanning dependencies of target mpi4py.MPI

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/ColorTable.cxx.o

[ 3%] Linking CXX shared library …/…/…/…/lib/libprotobuf-lite.so

[ 3%] Building C object VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/src/MPI.c.o

[ 3%] Built target protobuf-lite

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/CoordinateSystem.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/generated_message_reflection.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/message.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/reflection_ops.cc.o

In file included from /opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c:4:0:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx_PyCFunction_FastCall’:

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/serial/internal/ArrayManagerExecutionSerial.cxx.o

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163017:5: error: too many arguments to function ‘(struct PyObject * (*)(struct PyObject , struct PyObject * const, Py_ssize_t))meth’

return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);

^

In file included from /opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c:4:0:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx__ExceptionSave’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163228:19: error: ‘PyThreadState’ has no member named ‘exc_type’

*type = tstate->exc_type;

^

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlwriter.c.o

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163229:20: error: ‘PyThreadState’ has no member named ‘exc_value’

*value = tstate->exc_value;

^

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163230:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’

*tb = tstate->exc_traceback;

^

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx__ExceptionReset’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163237:22: error: ‘PyThreadState’ has no member named ‘exc_type’

tmp_type = tstate->exc_type;

^

…

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx_PyDict_Items’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164279:9: error: too many arguments to function ‘(struct PyObject * (*)(struct PyObject , struct PyObject * const, Py_ssize_t))__pyx_umethod_PyDict_Type_items.func’

return __Pyx_CallUnboundCMethod0(&__pyx_umethod_PyDict_Type_items, d);

^

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xpath.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xpointer.c.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/service.cc.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractDoubleEventPlayer.cxx.o

make[2]: *** [VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/src/MPI.c.o] Error 1

make[1]: *** [VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs…

mwestphal · March 7, 2019, 8:12am

A quick solution would be to disable MPI.

TimJim · March 7, 2019, 9:43am

I’ll try it without MPI. If I need to set it up with MPI, should it be picking up mpi4py in my install of python, or should it only pick up the ones that came in the VTK/ThirdParty directory? Thanks.

TimJim · March 8, 2019, 6:45am

Hi, sorry. I tried to run make without MPI and I’m seeing some linking issues with QT5. Do I need to update my QT package?

Thanks and regards,

Tim

/home/tjim/anaconda3/lib/libQt5Widgets.so.5.9.7: undefined reference to `operator delete[](void*, unsigned long)@CXXABI_1.3.9’

/home/tjim/anaconda3/lib/libQt5Core.so.5.9.7: undefined reference to `__cxa_throw_bad_array_new_length@CXXABI_1.3.8’

/home/tjim/anaconda3/lib/libQt5Widgets.so.5.9.7: undefined reference to `operator delete(void*, unsigned long)@CXXABI_1.3.9’

collect2: error: ld returned 1 exit status

make[2]: *** [bin/TestingDemo] Error 1

make[1]: *** [ThirdParty/QtTesting/vtkqttesting/Examples/CMakeFiles/TestingDemo.dir/all] Error 2

make: *** [all] Error 2

mwestphal · March 8, 2019, 9:17am

Are you bulding Qt within the superbuild or are you using an external Qt5 ?

TimJim · March 8, 2019, 10:25am

Based on your previous message, I’m not using the superbuild any more. Is there a QT package available inside? It seems that it’s picking up my anaconda installation QT: ~/anaconda3/lib/cmake/Qt5*

Are these not fully featured? Should I install QT 5 separately?

Thanks and regards.

mwestphal · March 8, 2019, 10:49am

Sorry, I mixed up answers.

You definitely need to install Qt, I would suggest using Qt 5.9.7 :
https://download.qt.io/archive/qt/5.9/5.9.7/

eric · March 8, 2019, 12:47pm

Hi,

Maybe this could help. In order to compile paraview 5.6.0 with python 3.6.2 (own build),
I had to change in the source directory, paraview/5.6.0/VTK/CMake/vtkWrapSettings.cmake the following:
L5: replace set(VTK_PYTHON_VERSION 2 CACHE STRING with
set(VTK_PYTHON_VERSION 3 CACHE STRING)

I also used the following cmake flags (for python) for the build:
[…]
-DPYTHON_LIBRARY=$PATH/python/3.6.2/lib/libpython3.6m.so.1.0 \
-DPYTHON_EXECUTABLE=$PATH/python/3.6.2/bin/python3
-DPYTHON_VERSION=3
[…]
(I also used Qt 5.9.3)

It seems to work fine for me.

Regards,
Eric