Compiling headless ParaView with OSMesa

robertsawko · November 11, 2021, 4:06pm

I am playing with in-situ visualisation and I have made some progress with EGL on GPU nodes, but would now like to enable it on every CPU-only nodes as well. I believe this should be possible with the use of OSMesa, but I am getting a little confused about dependencies. My main question is whether it’s possible to build a fully headless ParaView with OSMesa i.e. without any reference to X11 or other display server?

I have looked at @mwestphal tutorial, but I see that meson has an option -Dplatforms='x11'. I have also looked at this page in the documentation and I infer that I will need VTK_USE_X=OFF and VTK_OPENGL_HAS_OSMESA=ON. I am assuming this to be possible as there is a pre-compiled headless server with OSMesa on PV download page but please advise.

I’ve also got two questions or notes about that doc page:

VTK_OPENGL_HAS_OSMESA: When ON, implies that ParaView can use OSMesa to support headless modes of operation.

Is that meant to be headless or just offscreen? Or is “Off-screen rendering in Mesa” actually headless according to the terminology?

All combinations of above options can be turned on or off independently except that presently VTK_OPENGL_HAS_EGL and VTK_OPENGL_HAS_OSMESA are mutually exclusive i.e. only one of the two can be ON at the same time. This is because the current version of Mesa (17.1.5) doesn’t support EGL for OpenGL, it’s only supported for OpenGL-ES.

Is this still true? I was looking at Mesa 20+ and I think it now supports EGL.

mwestphal · November 11, 2021, 4:40pm

Offscreen mesa require a XOrg server.
Headless osmesa does not require a XOrg server.

Mesa 20 now supports EGL but VTK/ParaView does not yet support Mesa + EGL.

I am assuming this to be possible as there is a pre-compiled headless server with OSMesa on PV download page but please advise.

Of course this is possible, just build paraview and osmesa using the paraview superbuild, this is the easiest way forward: https://gitlab.kitware.com/paraview/paraview-superbuild

robertsawko · November 18, 2021, 3:15pm

Thanks. I started working with superbuild, but I am already getting a little confused. I have three questions

Do I understand correctly that options ENABLE_something mean that something is not to be built by superbuild. However if something is already built is it possible to point to an existing installation? For instance Python3 and MPI are already on this cluster, but I couldn’t succeed with the configuration until I switched it ON.
I don’t understand differences between editions, specifically CATALYST and CATALYST_RENDERING. What does it mean for Catalyst or Paraview not to have rendering capabilities?
Can you confirm that I use download-all correctly. I’ve done it like this:

ccmake \
  ...
  -DSUPERBUILD_OFFLINE_BUILD=ON \
  ../paraview-superbuild
make download-all
# submit a compilation job to compute

My current cmake invocation for future reference:

cmake \
    -DPARAVIEW_BUILD_EDITION=CATALYST_RENDERING \
    -DSUPERBUILD_OFFLINE_BUILD=ON \
    -DENABLE_mpi=ON \
    -DENABLE_python3=ON \
    -DENABLE_osmesa=ON \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
    ../paraview-superbuild

mwestphal · November 18, 2021, 3:17pm

No ENABLE_something means that pvsb will try to build and use it. ENABLE_something + USE_SYSTEM_something, means that pvsb will only try to use it from the system and not building it.

You 3. seems correct.

robertsawko · November 19, 2021, 2:57pm

Thanks, again. Mathieu. I think I am very close to getting it work. My non-system Python and non-MPI config builds fine, but now I am trying with system Python and MPI and it gets past a configuration stage but breaks with

[ 20%] Performing build step for 'pythonsetuptools'
cd /lustre/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/pythonsetuptools/src && /lustre/scafellpike/local/apps/hierarchy/core/cmake/3.21.1/bin/cmake -P /lustre/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/sb-pythonsetuptools-build.cmake
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    import setuptools
  File "/netfs/smain01/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/pythonsetuptools/src/setuptools/__init__.py", line 19, in <module>
    from setuptools.dist import Distribution
  File "/netfs/smain01/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/pythonsetuptools/src/setuptools/dist.py", line 34, in <module>
    from setuptools import windows_support
  File "/netfs/smain01/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/pythonsetuptools/src/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/lustre/scafellpike/local/HT04121/aas01/shared/apps/compiler/gcc/6.5.0/python/3.8.12/lib/python3.8/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
CMake Error at /lustre/scafellpike/local/HT04121/aas01/shared/software/paraview-superbuild/build-alt/superbuild/sb-pythonsetuptools-build.cmake:47 (message):
  Failed with exit code 1

I am tempted to just give up on system Python and have a separate Python in ParaView. We can always disable loading of other Pythons through modules, but EGL PV built against the same Python so I just wanted to be consistent.

My current build call for reference:

cmake \
    -DPARAVIEW_BUILD_EDITION=CATALYST_RENDERING \
    -DSUPERBUILD_OFFLINE_BUILD=ON \
    -DENABLE_python3=ON \
    -DUSE_SYSTEM_python3=ON \
    -DENABLE_mpi=ON \
    -DUSE_SYSTEM_mpi=ON \
    -DENABLE_osmesa=ON \
    -DENABLE_llvm=ON \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
    ../paraview-superbuild \
    && make download-all \
   # Job submission with parallel make

robertsawko · November 19, 2021, 3:04pm

And sorry, one more slightly unrelated question. Is there a way to see what options for each software were used for a given superbuild configuration? If I want to inspect step-by-step how software stack was built because, can I do that?

mwestphal · November 19, 2021, 3:06pm

No idea, maybe @ben.boeckel can help.

Is there a way to see what options for each software were used for a given superbuild configuration? If I want to inspect step-by-step how software stack was built because, can I do that?

You can read the CMakeCache.txt but it is very verbose, nothing more then that I’m afraid.

robertsawko · November 19, 2021, 3:17pm

Actually a quick search suggests it’s libffi and libffi-devel that have something to do with interfacing between functions of different languages. It may take me a while to get this installed on this computer, but I will see what I can do.

ben.boeckel · November 19, 2021, 11:42pm

You can look at the superbuild/sb-*-configure.cmake (there are also -build.cmake and -install.cmake files, but these are fairly boring usually) files in the build directory.

As for Python, yes, it appears that the Python you have is not enough to build some things (such as setuptools apparently?).

robertsawko · November 24, 2021, 10:34pm

Thanks, a lot.

I have one final, I think, set of questions about superbuild. I noticed that I didn’t get the include/, share/ and *.cmake file in my installation. Unfortunately, I need some of them as I am building my Catalyst adaptors so will need to compile whilst pointing to existing ParaView.

For comparison the EGL version that I installed manually has the follow structure

egl:
bin  include  lib64  share

osmesa:
bin  lib

I am now rerunning the build with

DCMAKE_BUILD_TYPE_paraview=RelWithDebInfo

Is there anything else I need to be aware of?

ben.boeckel · November 25, 2021, 3:14pm

What do you mean by “installation”? The results of make install at the superbuild level? If you want that to generate headers and such, you need the paraviewsdk project enabled.

robertsawko · November 25, 2021, 3:50pm

Thanks, Ben and Mathieu. That is exactly what I have meant.

I am recompiling and if I get it right this time, I will post my final cmake/make invocations.

If I ever get my project closer to where the proposal says it will be and if you happen to be at the same conference I should at the very minimum take you out for a drink!

robertsawko · November 26, 2021, 3:54pm

Setting ENABLE_paraviewsdk worked for PV and VTK, but I am sorry, I fear that only generated more questions…

Is it possible get headers of all the packages to be also installed under current configuration? Currently a CMake of my Catalyst adaptopr is failing because it cannot find OSMesa headers which are not installed on this machine.
Do I need to set CMAKE_PREFIX_PATH or CMAKE_MODULE_PATH and I do need to cover each subdirectory?
What does DSUPERBUILD_DEBUG_CONFIGURE_STEPS=ON do?

Here’s my current invocation.

cmake \
    -DCMAKE_BUILD_TYPE_paraview=RelWithDebInfo \
    -DPARAVIEW_BUILD_EDITION=CANONICAL \
    -DSUPERBUILD_OFFLINE_BUILD=ON \
    -DENABLE_python3=ON \
    -DENABLE_mpi=ON \
    -DUSE_SYSTEM_mpi=ON \
    -DENABLE_osmesa=ON \
    -DENABLE_llvm=ON \
    -DENABLE_paraviewsdk=ON \
    -DSUPERBUILD_DEBUG_CONFIGURE_STEPS=ON \
    -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
    ../paraview-superbuild \
    && make download-all \
    && make install

ben.boeckel · November 28, 2021, 1:12am

If you’re building on a single machine, I would recommend building against the $superbuild_build_dir/install directly rather than the artifacts of any of the superbuild’s install mechanisms.