Complete compilation of Paraview (Readers+OSmesa+Python)

Dear support,

I am trying to compile paraview to have a solid version in Linux because using the uncompiled version gives me problems with either the readers, the python libraries or the GPU drivers (I would like to use Mesa since I am mostly using CPU rendering in my project).

Here I leave the steps and commands that I followed for the compilation:

Declaration of my installation path

#--------------------------------------------#
# Installation path
#--------------------------------------------#
InstallPath='/home/myUsername/Documents/Paraview_build'

Download and compile LLVM source

#--------------------------------------------#
# Part 1: Install LLVM source
#--------------------------------------------#

# SOURCE: https://www.paraview.org/Wiki/ParaView/ParaView_And_Mesa_3D

# Download the latest LLVM source
curl -L -O https://github.com/llvm/llvm-project/releases/download/llvmorg-12.0.1/llvm-12.0.1.src.tar.xz

# Extract
tar -xvf llvm-12.0.1.src.tar.xz

# Configure
mkdir llvm-12.0.1.bld
cd llvm-12.0.1.bld
cmake                                           \
  -DCMAKE_BUILD_TYPE=Release                    \
  -DCMAKE_INSTALL_PREFIX=${InstallPath}/llvm    \
  -DLLVM_BUILD_LLVM_DYLIB=ON                    \
  -DLLVM_ENABLE_RTTI=ON                         \
  -DLLVM_INSTALL_UTILS=ON                       \
  -DLLVM_TARGETS_TO_BUILD=X86                   \
  ../llvm-12.0.1.src

# Build and install
make -j8
make install

# Set your dependencies in your .basrhc
export PATH=$PATH:${InstallPath}/llvm/bin
cd ..

Then I download and install the Mesa source

#--------------------------------------------#
# Part 2: Install MESA source
#--------------------------------------------#

# Download and extract the latest Mesa source
curl -L -O https://archive.mesa3d.org//mesa-21.3.1.tar.xz
tar -xvf mesa-21.3.1.tar.xz

# Configure Mesa
mkdir mesa-21.3.1-build
cd mesa-21.3.1-build

meson                          \
  ../mesa-21.3.1               \
  --buildtype=release          \
  --prefix=${InstallPath}/mesa \
  -Dvulkan-drivers=            \
  -Ddri-drivers=               \
  -Dgallium-vdpau=disabled     \
  -Dgallium-xvmc=disabled      \
  -Dgallium-omx=disabled       \
  -Dgallium-va=disabled        \
  -Dgallium-xa=disabled        \
  -Dgallium-nine=false         \
  -Dgallium-opencl=disabled    \
  -Dbuild-tests=false          \
  -Degl=disabled               \
  -Dgbm=disabled               \
  -Dglx=disabled               \
  -Dplatforms=                 \
  -Dglvnd=false                \
  -Dosmesa=true                \
  -Dopengl=true                \
  -Dgles1=disabled             \
  -Dgles2=disabled             \
  -Dshared-glapi=enabled       \
  -Dllvm=enabled               \
  -Dshared-llvm=enabled        \
  -Dgallium-drivers=swrast,swr

ninja
ninja install

#./configure --prefix=/home/cjimenez2/Documents/Paraview_build --enable-osmesa --with-osmesa-bits=8 ac_cv_path_LLVM_CONFIG=llvm-config-x.x

Then I download the required dependencies

#--------------------------------------------#
# Part 3: Download dependencies for Paraview
#--------------------------------------------#

#SOURCE: https://gitlab.kitware.com/paraview/paraview/blob/master/Documentation/dev/build.md

# Dependencies
sudo apt-get install git cmake build-essential libgl1-mesa-dev libxt-dev qt5-default libqt5x11extras5-dev libqt5help5 qttools5-dev qtxmlpatterns5-dev-tools libqt5svg5-dev python3-dev python3-numpy libopenmpi-dev libtbb-dev ninja-build openmpi-bin

# Environtment
export PATH=$PATH:/usr/bin/

And finally I download and install Paraview version 5.9.1

#--------------------------------------------#
# Part 4: Download and compile Paraview
#--------------------------------------------#

# To build a specific ParaView version, eg: v5.9.1, please run the following commands in a terminal while replacing "tag" by the version you want to build

git clone https://gitlab.kitware.com/paraview/paraview.git

cd paraview
git checkout v5.9.1
git submodule update --init --recursive
cd ..

#--------------------------------------------
# OSMESA compilation configuration
cmake \
-GNinja \
-DPARAVIEW_USE_PYTHON=ON \
-DPARAVIEW_USE_MPI=ON \
-DVTK_SMP_IMPLEMENTATION_TYPE=TBB \
-DPARAVIEW_ENABLE_VISITBRIDGE=ON \
-DCMAKE_BUILD_TYPE=Release ./paraview \
-DPARAVIEW_BUILD_QT_GUI=OFF                                  \
-DVTK_USE_X=OFF                                              \
-DOPENGL_INCLUDE_DIR=IGNORE                                  \
-DOPENGL_xmesa_INCLUDE_DIR=IGNORE                            \
-DOPENGL_gl_LIBRARY=IGNORE                                   \
-DOSMESA_INCLUDE_DIR=${InstallPath}/mesa-21.3.1/include      \
-DOSMESA_LIBRARY=${InstallPath}/mesa/lib/x86_64-linux-gnu/libOSMesa.so        \
-DVTK_OPENGL_HAS_OSMESA=ON                                   \
-DVTK_USE_OFFSCREEN=OFF 
                                 
# Compilation command
ninja
#--------------------------------------------#

However after the cmake command I get the following warning:

-- Enabled modules: VTK(155), ParaView(48 + 12)
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    OPENGL_INCLUDE_DIR
    OPENGL_gl_LIBRARY
    OPENGL_xmesa_INCLUDE_DIR
    VTK_USE_OFFSCREEN

And the compilation crashes when using ninja after the following mensaje:

[1592/20515] Building C object VTK/ThirdParty/glew/vtkglew/CMakeFiles/glew.dir/src/glew.c.o
FAILED: VTK/ThirdParty/glew/vtkglew/CMakeFiles/glew.dir/src/glew.c.o 
/usr/bin/cc -DGLEW_NO_GLU -DGLEW_OSMESA -Dglew_EXPORTS -IVTK/ThirdParty/glew/vtkglew -Iparaview/VTK/ThirdParty/glew/vtkglew -Iparaview/VTK/ThirdParty/glew/vtkglew/include -isystem VTK/ThirdParty/glew -isystem paraview/VTK/ThirdParty/glew -isystem mesa-21.3.1/include -O3 -DNDEBUG -fPIC -MD -MT VTK/ThirdParty/glew/vtkglew/CMakeFiles/glew.dir/src/glew.c.o -MF VTK/ThirdParty/glew/vtkglew/CMakeFiles/glew.dir/src/glew.c.o.d -o VTK/ThirdParty/glew/vtkglew/CMakeFiles/glew.dir/src/glew.c.o   -c paraview/VTK/ThirdParty/glew/vtkglew/src/glew.c
paraview/VTK/ThirdParty/glew/vtkglew/src/glew.c:41:12: fatal error: GL/osmesa.h: No such file or directory
   41 | #  include <GL/osmesa.h>
      |            ^~~~~~~~~~~~~
compilation terminated.
[1605/20515] Building CXX object VTK/ThirdParty/jsoncpp/vtkjsoncpp/CMakeFiles/jsoncpp.dir/jsoncpp.cpp.o
ninja: build stopped: subcommand failed.

I think that the problem is linked with the fact that I should ignore the GL libraries but the flags that I put in the cmake command were not read.

Is there any updated guide to compile paraview almost from scratch?

Thank you in advance!

Hi @Captain_Convergence

You are looking for the ParaView-Superbuild: https://gitlab.kitware.com/paraview/paraview-superbuild/

Best,

Dear Mathieu,

Thank you so much for your message. I dowloaded the super-build and I am trying to compile it with OSMESA (llvm and mesa are already compiled in my computer). I am using the following flags:

cmake \
-GNinja \
-DPARAVIEW_USE_PYTHON=ON \
-DPython3_EXECUTABLE:FILEPATH=/usr/bin/python3 \
-DPARAVIEW_USE_MPI=ON \
-DVTK_SMP_IMPLEMENTATION_TYPE=TBB \
-DPARAVIEW_ENABLE_VISITBRIDGE=ON \
-DOSMESA_INCLUDE_DIR=${InstallPath}/mesa-21.3.1/include \
-DOSMESA_LIBRARY=${InstallPath}/mesa/lib/x86_64-linux-gnu/libOSMesa.so \
-DCMAKE_BUILD_TYPE=Release ./paraview-superbuild

However it crashes because python 2 or 3 have not been enabled:

-- Building projects: cxx11, nlohmannjson, python, pybind11, paraview
CMake Error at superbuild/projects/python.cmake:2 (message):
  Python is enabled, but neither Python2 nor Python3 has been enabled.
Call Stack (most recent call first):
  superbuild/cmake/SuperbuildMacros.cmake:983 (include)
  superbuild/CMakeLists.txt:159 (superbuild_process_dependencies)

I also read the post below because some people were experiencing the same issue and you recommended using the flag " -D ENABLE_python3":

However for the current super-build version, that cmake flag is not recognized, I also tried:

-D ENABLE_python3 \
-DENABLE_python3 \
-DPARAVIEW_ENABLE_python3 \

But I cannot start the compilation.

Hi @Captain_Convergence

There is no reason to specify such cmake options, just enable what you need and the superbuild will download and build it for you. This is literally the point of the superbuild, you do not need to build the dependencies yourself.

So in your case, this should be enough:

cmake \
-GNinja \
-DENABLE_osmesa=ON
-DENABLE_python3=ON
-DENABLE_mpi=ON
-DENABLE_paraview=ON
-DCMAKE_BUILD_TYPE=Release
../paraview-superbuild

Also it looks like you are building an old version of the superbuild, is that intended ?

Best,

Dear Mathieu,

I tried with the flags that you mentioned and it seems to have worked. However I don’t see the “paraview” executable in the “install/bin” folder:

I can see that the rest of utilities have been succesfully compiled. How can I call the Paraview interface to directly work with the GUI?

You need to add ENABLE_qt5

I’m afraid ENABLE_osmesa and ENABLE_qt5 are incompatible. You can’t build ParaView Qt interface with osmesa at the same time.

@Captain_Convergence , why do you need osmesa rendering if you want to run ParaView also ?

The reason for this is that I need to use ParaView in parallel to compute some field in a reasonable time but I noticed that I have problems when using multiple CPUs, see my previous post here to now more about the issue:

However, I ran by chance in our HPC the ParaView version with osmesa using multiple-CPUs and I didn’t have the problem so now I believe that the solution to all my issues is to use a compiled version of ParaView using osmesa since I normally do CPU-based rendering.

I tried to compile the GUI using:

cmake \
-GNinja \
-DENABLE_qt5=ON \
-DENABLE_python3=ON \
-DENABLE_mpi=ON \
-DENABLE_paraview=ON \
-DCMAKE_BUILD_TYPE=Release ./paraview-superbuild

But I have the following compilation error linked with the LLVM installation:

It works for the -DENABLE_osmesa=ON compilation but it fails when I set -DENABLE_qt5=ON. Do I need to set any additional flag to correctly compile Paraview with the GUI?

I set -DENABLE_qt5=ON. Do I need to set any additional flag to correctly compile Paraview with the GUI?

Yes, we do not build Qt anymore, use the one from your system, but

If for some reason the binary release pvserver do not work for you, then use your osmesa build for the server and the binary release for the client.

So if I understand correctly, I can source the binary files from the latest Paraview Version (nightly/Master Version from https://www.paraview.org/download/) to use the GUI and then use pvserver to connect to the built version with OSMESA right? Something like this:

export PATH=/homeDir/ParaView-master-5.11.0/bin:$PATH
export PATH=/homeDir/ParaviewSuperbuild_OSMESA/install/bin:$PATH
alias svr='mpirun -np 6 pvserver --force-offscreen-rendering --server-port=11111'

I can connect to the SuperBuild server from the master version but then I have the problem regarding the readers since I haven’t compiled the version 5.11. Also I didn’t add -DPARAVIEW_ENABLE_VISITBRIDGE=ON \ during the superbuild compilation (since I thought that it would be included by default) so maybe that’s why the readers are not available in the pvserver. However adding the aforementioned flag doesn’t seem to work for the superbuild:

image

Why do you want to use master ? Juste use the last release, on both superbuild and binary.

I manage to build the 5.10.1 tag to completion on Ubuntu 20.04 (no discrete GPU)

However, when I try to do a ninja install I get the following errors

Copying /home/nyue/projects/ParaView/superbuild/qt-build/install/lib/catalyst/libcatalyst-paraview.so ==> lib/catalyst
CMake Error at cmake_install.cmake:1478 (message):
  Failed to install
  /home/nyue/projects/ParaView/superbuild/qt-build/install/lib/catalyst/libcatalyst-paraview.so:


  objdump:
  '/home/nyue/projects/ParaView/superbuild/qt-build/install/lib/catalyst/libcatalyst-paraview.so':
  No such file

  Traceback (most recent call last):

    File "/home/nyue/projects/ParaView/superbuild/paraview-superbuild_git/superbuild/cmake/scripts/fixup_bundle.unix.py", line 750, in <module>
      main(sys.argv[1:])
    File "/home/nyue/projects/ParaView/superbuild/paraview-superbuild_git/superbuild/cmake/scripts/fixup_bundle.unix.py", line 742, in main
      _install_binary(main_exe, is_excluded, bundle_dest, opts.libdir, installed, manifest, opts.source, dry_run=opts.dry_run, look_for_symlinks=opts.has_symlinks)
    File "/home/nyue/projects/ParaView/superbuild/paraview-superbuild_git/superbuild/cmake/scripts/fixup_bundle.unix.py", line 637, in _install_binary
      shutil.copy(binary.path, app_dest)
    File "/home/nyue/projects/ParaView/superbuild/qt-build/install/lib/python3.9/shutil.py", line 418, in copy
      copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/home/nyue/projects/ParaView/superbuild/qt-build/install/lib/python3.9/shutil.py", line 264, in copyfile
      with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:

  FileNotFoundError: [Errno 2] No such file or directory:
  '/home/nyue/projects/ParaView/superbuild/qt-build/install/lib/catalyst/libcatalyst-paraview.so'




make: *** [Makefile:120: install] Error 1
CMake Error at superbuild/cmake_install.cmake:66 (message):
  Failed to install the paraview/TGZ package.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)


FAILED: CMakeFiles/install.util 
cd /home/nyue/projects/ParaView/superbuild/qt-build && /home/nyue/systems/cmake/cmake-3.24.3-linux-x86_64/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

Is catalyst mandatory ?

cmake \
    -D ENABLE_qt5=ON \
    -D ENABLE_python3=ON \
    -D ENABLE_mpi=ON \
    -D ENABLE_paraview=ON \
    -D CMAKE_INSTALL_PREFIX=$HOME/systems/ParaView/5.10.1 \
    -D CMAKE_BUILD_TYPE=Release \
    -G Ninja \
    ../paraview-superbuild_git

Hi

the superbuild do not use not support the install target.

Best,

The superbuild documentation about installing should be updated to reflect that install is not use nor supported.

What is the recommend step post-superbuild to deploy and package if install target is not the way to do it ?

See the packaging part: https://gitlab.kitware.com/paraview/paraview-superbuild#packaging

This installing parts looks like old doc that did not get removed, wdyt @ben.boeckel ?

False. It does. The documentation is up-to-date.

Well, this path is suspicious. Why is it looking for ParaView’s Catalyst implementation under Qt’s build tree?

I must remember it wrong then.

Hi @ben.boeckel

I just so happen to name my out of source build directory to be called qt-build, I am also planning to have an osmesa-build directory so that I don’t mix them up (qt and osmesa options cannot be mixed).

Or are you saying that libcatalyst should be a totally different piece of software and not something that should be built via superbuild ?

I am not familiar with catalyst, I don’t use with Paraview and hence have no experience with it.

Cheers

It’s an HPC thing normally, but it should be fine (it just runs the packaging script with the install destination as the place to put things instead of where CPack expects them.

Ah, ok, I see now. It just happened to look like qt/build.

If you don’t need catalyst, you can either disable it in the superbuild (ENABLE_catalyst=OFF) and possibly see where in projects/unix/paraview.bundle.unix.cmake it comes from (near line 260 or so).

2 Likes