Superbuild with Anaconda Python 3.7.1

ParaView Support
13

Thanks, Mathieu. I tried to make the build but it seems that I’m getting another issue with mpi4py. It seems to try to grab the version in my python directory and also in the Paraview directories. Can you give me some tips on how to fix this? Thank you. Here are are a few lines of the errors:

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/message_lite.cc.o

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/init.py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/main.py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include/mpi4py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/lib-pmpi’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/init.py’…

Compiling ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/main.py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/include/mpi4py’…

Listing ‘/opt/paraview/ParaView-5.6.0_build/lib/python3.7/site-packages/mpi4py/lib-pmpi’…

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/ArrayRangeCompute.cxx.o

[ 3%] Built target mpi4py

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/CellSetExplicit.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/repeated_field.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlsave.c.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pq3DViewEventPlayer.cxx.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pq3DViewEventTranslator.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/wire_format_lite.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlschemas.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlschemastypes.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlstring.c.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/coded_stream.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/zero_copy_stream.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protobuf-lite.dir/google/protobuf/io/zero_copy_stream_impl_lite.cc.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlunicode.c.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractActivateEventPlayer.cxx.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractBooleanEventPlayer.cxx.o

Scanning dependencies of target mpi4py.MPI

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/ColorTable.cxx.o

[ 3%] Linking CXX shared library …/…/…/…/lib/libprotobuf-lite.so

[ 3%] Building C object VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/src/MPI.c.o

[ 3%] Built target protobuf-lite

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/CoordinateSystem.cxx.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/generated_message_reflection.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/message.cc.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/reflection_ops.cc.o

In file included from /opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c:4:0:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx_PyCFunction_FastCall’:

[ 3%] Building CXX object VTK/ThirdParty/vtkm/vtk-m/vtkm/cont/CMakeFiles/vtkm_cont.dir/serial/internal/ArrayManagerExecutionSerial.cxx.o

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163017:5: error: too many arguments to function ‘(struct PyObject * (*)(struct PyObject , struct PyObject * const, Py_ssize_t))meth’

return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);

^

In file included from /opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c:4:0:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx__ExceptionSave’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163228:19: error: ‘PyThreadState’ has no member named ‘exc_type’

*type = tstate->exc_type;

^

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xmlwriter.c.o

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163229:20: error: ‘PyThreadState’ has no member named ‘exc_value’

*value = tstate->exc_value;

^

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163230:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’

*tb = tstate->exc_traceback;

^

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx__ExceptionReset’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:163237:22: error: ‘PyThreadState’ has no member named ‘exc_type’

tmp_type = tstate->exc_type;

^

…

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx_PyDict_Items’:

/opt/paraview/ParaView/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164279:9: error: too many arguments to function ‘(struct PyObject * (*)(struct PyObject , struct PyObject * const, Py_ssize_t))__pyx_umethod_PyDict_Type_items.func’

return __Pyx_CallUnboundCMethod0(&__pyx_umethod_PyDict_Type_items, d);

^

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xpath.c.o

[ 3%] Building C object VTK/ThirdParty/libxml2/vtklibxml2/CMakeFiles/vtklibxml2.dir/xpointer.c.o

[ 3%] Building CXX object ThirdParty/protobuf/vtkprotobuf/src/CMakeFiles/protoc_compiler.dir/google/protobuf/service.cc.o

[ 3%] Building CXX object ThirdParty/QtTesting/vtkqttesting/CMakeFiles/QtTesting.dir/pqAbstractDoubleEventPlayer.cxx.o

make[2]: *** [VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/src/MPI.c.o] Error 1

make[1]: *** [VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeFiles/mpi4py.MPI.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs…

14

A quick solution would be to disable MPI.

15

I’ll try it without MPI. If I need to set it up with MPI, should it be picking up mpi4py in my install of python, or should it only pick up the ones that came in the VTK/ThirdParty directory? Thanks.

16

Hi, sorry. I tried to run make without MPI and I’m seeing some linking issues with QT5. Do I need to update my QT package?

Thanks and regards,

Tim

/home/tjim/anaconda3/lib/libQt5Widgets.so.5.9.7: undefined reference to `operator delete[](void*, unsigned long)@CXXABI_1.3.9’

/home/tjim/anaconda3/lib/libQt5Core.so.5.9.7: undefined reference to `__cxa_throw_bad_array_new_length@CXXABI_1.3.8’

/home/tjim/anaconda3/lib/libQt5Widgets.so.5.9.7: undefined reference to `operator delete(void*, unsigned long)@CXXABI_1.3.9’

collect2: error: ld returned 1 exit status

make[2]: *** [bin/TestingDemo] Error 1

make[1]: *** [ThirdParty/QtTesting/vtkqttesting/Examples/CMakeFiles/TestingDemo.dir/all] Error 2

make: *** [all] Error 2

17

Are you bulding Qt within the superbuild or are you using an external Qt5 ?

18

Based on your previous message, I’m not using the superbuild any more. Is there a QT package available inside? It seems that it’s picking up my anaconda installation QT: ~/anaconda3/lib/cmake/Qt5*

Are these not fully featured? Should I install QT 5 separately?

Thanks and regards.

19

Sorry, I mixed up answers.

You definitely need to install Qt, I would suggest using Qt 5.9.7 :
https://download.qt.io/archive/qt/5.9/5.9.7/

20

Hi,

Maybe this could help. In order to compile paraview 5.6.0 with python 3.6.2 (own build),
I had to change in the source directory, paraview/5.6.0/VTK/CMake/vtkWrapSettings.cmake the following:
L5: replace set(VTK_PYTHON_VERSION 2 CACHE STRING with
set(VTK_PYTHON_VERSION 3 CACHE STRING)

I also used the following cmake flags (for python) for the build:
[…]
-DPYTHON_LIBRARY=$PATH/python/3.6.2/lib/libpython3.6m.so.1.0 \
-DPYTHON_EXECUTABLE=$PATH/python/3.6.2/bin/python3
-DPYTHON_VERSION=3
[…]
(I also used Qt 5.9.3)

It seems to work fine for me.

Regards,
Eric

21

I forgot to mention that my comment concerned the standalone version of Paraview 5.6.0 and not the superbuild. I did not use MPI support.
/Eric

22

Hi Eric, Mathieu,

Thanks for your tips - I’m still having a few issues. After specifying the correct Python library and executable paths, it not seems to interfere with my QT install. Although I have specified the QT paths to my system install, it seems that ccmake tries to hook in the Python QT libraries. I tried removing Anaconda from my path to make sure it wasn’t finding the libraries from PATH but it still seems to pick it up. The message I get on ccmake ‘generate’ is attached below. Any thoughts or suggestions would be appreciated!

Many thanks and kind regards,

Tim

CMake Warning at VTK/CMake/vtkModuleMacros.cmake:575 (add_library):

Cannot generate a safe runtime search path for target pqWidgets because

files in some directories may conflict with libraries in implicit

directories:

runtime library [libQt5Network.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Help.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Widgets.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5X11Extras.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Gui.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Sql.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Core.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

Some of these libraries may not be found correctly.

Call Stack (most recent call first):

VTK/CMake/vtkModuleMacros.cmake:660 (vtk_add_library)

Qt/Widgets/CMakeLists.txt:269 (vtk_module_library)

23

Hi Tim,
Here are the cMake flags I used for Qt (5.9.3) if it can help
-DPARAVIEW_BUILD_QT_GUI=ON
-DPARAVIEW_QT_VERSION=5 -DQt5_DIR=$MYPATH/qt/5.9.3/lib/cmake/Qt5
Note also that in my previous comment I made a typo $PATH should be $MYPATH (where your python installation is located). Sorry for that.
Regards
Eric

24

Hi Eric,

Thanks! Does this mean you installed QT5 within your python install using pip or similar?

Thanks,

Tim

25

Tim,
No, I installed Qt separately. You also need to make sure that your $PATH and $LD_LIBRARY_PATH are correct so that you pick up the correct Qt, python, etc. when you run cmake.
Eric

26

For the issue you are having, I would recommend checking the CMakeCache.txt for any remaining mention of your anaconda Qt install, and also setting LD_LIBRARY_PATH and PATH correctly.

27

Thanks for the tips. I installed all the QT libraries and set all the flags. I tried again starting from a fresh directory and adding flags one at a time. It all seems to be fine setting the QT flags until I add my python directory - from then, ‘generate’ results in messages similar to below. I’ve made sure that the paths are clear and that the only mentions of my python install ‘/home/tjim/anaconda3/envs/pv/’ are in the PYTHON_LIBRARY and PYTHON_INCLUDE_DIR variables. I’m not sure what to try next - I could remove the conflicting libraries from my python install, but that will kill Matplotlib or other graphical libraries I think. Do you have any other ideas I might be able to try? Many thanks.

CMake Warning at VTK/CMake/vtkModuleMacros.cmake:575 (add_library):

Cannot generate a safe runtime search path for target pqWidgets because

files in some directories may conflict with libraries in implicit

directories:

runtime library [libQt5Network.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Help.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Widgets.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5X11Extras.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Gui.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Sql.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

runtime library [libQt5Core.so.5] in /usr/lib64 may be hidden by files in:

/home/tjim/anaconda3/envs/pv/lib

Some of these libraries may not be found correctly.

Call Stack (most recent call first):

VTK/CMake/vtkModuleMacros.cmake:660 (vtk_add_library)

Qt/Widgets/CMakeLists.txt:269 (vtk_module_library)

28

I’m running into the same issue as OP regarding a failing build with MPI, python 3.7, and mpi4py. I’ve tested the same configuration on a similar system with python 3.6 and that works as expected - I think this is an issue with mpi4py w.r.t. python 3.7.

...
...
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164083:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
     tstate->exc_type = *type;
             ^~~~~~~~
             curexc_type
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164084:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     tstate->exc_value = *value;
             ^~~~~~~~~
             curexc_value
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164085:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     tstate->exc_traceback = *tb;
             ^~~~~~~~~~~~~
             curexc_traceback
In file included from ../VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c:4:
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c: In function ‘__Pyx_PyDict_Items’:
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:1851:83: error: too many arguments to function ‘(PyObject * (*)(PyObject *, PyObject * const*, Py_ssize_t))__pyx_umethod_PyDict_Type_items.func’
               (PY_VERSION_HEX >= 0x030600B1 && (cfunc)->flag == METH_FASTCALL ?  (*(__Pyx_PyCFunctionFast)(cfunc)->func)(self, &PyTuple_GET_ITEM(__pyx_empty_tuple, 0), 0, NULL) :\
                                                                                  ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c:164279:16: note: in expansion of macro ‘__Pyx_CallUnboundCMethod0’
         return __Pyx_CallUnboundCMethod0(&__pyx_umethod_PyDict_Type_items, d);
                ^~~~~~~~~~~~~~~~~~~~~~~~~
[4902/19480] Building C object ThirdParty/cgns/vtkcgns/src/CMakeFiles/vtkcgns.dir/cgnslib.c.o
[4903/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkExtractPointCloudPiece.cxx.o
[4904/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkExtractPoints.cxx.o
[4905/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkFitImplicitFunction.cxx.o
[4906/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkGaussianKernel.cxx.o
[4907/19480] Building CXX object ParaViewCore/VTKExtensions/CGNSReader/CMakeFiles/vtkPVVTKExtensionsCGNSReader.dir/vtkCGNSReader.cxx.o
[4908/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkExtractEnclosedPoints.cxx.o
[4909/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkDensifyPointCloudFilter.cxx.o
[4910/19480] Building CXX object VTK/Filters/Points/CMakeFiles/vtkFiltersPoints.dir/vtkExtractSurface.cxx.o
ninja: build stopped: subcommand failed.

Here’s a copy of the cache file I use to preload the cmake build:

SET(BUILD_EXAMPLES ON CACHE BOOL "")
SET(PARAVIEW_ENABLE_FFMPEG ON CACHE BOOL "")
SET(PARAVIEW_ENABLE_PYTHON ON CACHE BOOL "")
SET(PARAVIEW_USE_MPI ON CACHE BOOL "")
SET(VTK_PYTHON_VERSION 3 CACHE STRING "")
SET(CMAKE_BUILD_TYPE "Release" CACHE STRING "")

find_program(CMAKE_C_COMPILER "gcc")
find_program(CMAKE_CXX_COMPILER "g++")
find_program(CMAKE_Fortran_COMPILER "gfortran")
find_program(MPI_C_COMPILER "mpicc")
find_program(MPI_CXX_COMPILER "mpicxx")
find_program(MPI_Fortran_COMPILER "mpifort")
find_program(PYTHON_EXECUTABLE "python3")

find_program(QT_QMAKE_EXECUTABLE "qmake-qt5")
find_program(QT_HELP_GENERATOR "qhelpgenerator-qt5")
find_program(QT_XMLPATTERNS_EXECUTABLE "xmlpatterns-qt5")

Copies of the resulting CmakeCache.txt file and log files are attached. Please advise if I should open up a new issue

CMakeCache.txt (188.9 KB)
log.cmake (40.0 KB)
log.make (590.0 KB)

29

The mpi4py generated using the Cython available at import time that is included in VTK 8.2.0 (which it appears you may be trying to compile) might not support those versions of Python. You could try doing this:

$EDITOR ThirdParty/mpi4py/update.sh
# change `tag="for/vtk"` to `tag="f79c82e4bd527a7770d1bd2a4fb53dab605678a2"`
ThirdParty/mpi4py/update.sh

with a newer Cython installed in case that gets you 3.7 support. Note that you need to be at the VTK root in order to run the script and it’s only really decently tested on Linux (there are known issues when running on macOS and Windows).

30

Hi Ben,

Thanks for getting back to me.

I tried changing the line to the git tag you listed but that doesn’t seem to work – that git hash doesn’t exist in the mpi4py, at least that’s what it looks like from the following error:

fatal: reference is not a tree: f79c82e4bd527a7770d1bd2a4fb53dab605678a2

The most recent commit that references that git hash is 3ad62583885dfe6f4e940cbda54e813ccc196b80, but that hash causes the same fatal error.

I also tried just running ./update.sh with the current for/vtk tag, and that resulted in at least building the mpi4py project, but also a merge clash

NOTE: All of this was done from a fresh git clone of the ParaView release branch (with associated submodules initialized)

Met vriendelijke groet,

REDstack BV

Chris Coutinho

Researcher/Data Analyst

Verzonden: maandag 6 mei 2019 21:11

31

I’ll push up a tag for that commit. It is for/vtk-20170419-2.0.0 instead of the raw hash.

32

Thanks @ben.boeckel, using that commit/tag works, and now I can build ParaView on my system with python 3.7.

Many thanks!