Building Paraview "from scratch" in order to use a custom plugin

ParaView Support
1

Hi all,

I’m not sure if this is a common request/issue (I’d assume it is?), but I desperately need help with building and installing paraview on a remote linux cluster.

I have spent just over two long days trying to build Paraview (specifically either v5.2.0 or v5.6.0) on a linux machine [Centos 7 ] . I’ve tried following the instructions on the paraview website, first on the old wiki- https://www.paraview.org/Wiki/ParaView:Build_And_Install

then here - https://gitlab.kitware.com/paraview/paraview/blob/master/Documentation/dev/build.md#linux

I have also tried the superbuild https://gitlab.kitware.com/paraview/paraview-superbuild/-/blob/master/README.md

So far, I have not been able to successfully install the plugin, and have only made it through the entire build process once (using the following set of commands):

git clone https://gitlab.kitware.com/paraview/paraview.git
cd paraview
git checkout v5.2.0
git submodule update --init --recursive

cp -r ~/ParaViewSyncIOReaderPlugin/ Plugins/
cd …

git clone --recursive https://gitlab.kitware.com/paraview/paraview-superbuild.git
cd paraview-superbuild
git fetch origin
git checkout v5.2.0
git submodule update

cmake -GNinja …/paraview
ninja

The cmake version I used was version 3.16.3

The ninja version was 1.10

I believe the Qt version was 4.8.7

I know the Qt version should be 5.9 or higher, and indeed the Qt version used in most tests was a freshly installed 5.9.0 version, but compilation would fail either around 94% (using make)

[ 94%] Building CXX object Qt/Components/CMakeFiles/pqComponents.dir/pqPresetToPixmap.cxx.o
[ 94%] Building CXX object Qt/Components/CMakeFiles/pqComponents.dir/pqPropertiesPanel.cxx.o
": internal error: 101004_111
compilation aborted for /usr/local/usrapps/iabolotn/paraview_5.2.0/paraview/Qt/Components/pqPresetToPixmap.cxx (code 4)
make[2]: *** [Qt/Components/CMakeFiles/pqComponents.dir/pqPresetToPixmap.cxx.o] Error 4
make[2]: *** Waiting for unfinished jobs…
make[1]: *** [Qt/Components/CMakeFiles/pqComponents.dir/all] Error 2
make: *** [all] Error 2

or 26%

[ 26%] Building CXX object VTK/IO/LSDyna/CMakeFiles/vtkIOLSDyna.dir/LSDynaFamily.cxx.o
[ 26%] Linking CXX shared library …/…/…/lib/libvtkIOLSDyna-pv5.6.so
[ 26%] Built target vtkIOLSDyna
make: *** [all] Error 2

“/usr/local/usrapps/iabolotn/try_10_paraview/paraview/VTK/ThirdParty/pegtl/vtkpegtl/include/tao/pegtl/parse.hpp”
instantiation of “bool tao::pegtl::parse<Rule,Action,Control,A,M,Input,States…>(Input &&, States &&…) [with Rule=MotionFX::CFG::Grammar, Action=Actions::CFG::action, Control=tao::pegtl::normal, A=tao::pegtl::apply_mode::ACTION, M=tao::pegtl::rewind_mode::REQUIRED, Input=tao::pegtl::read_input<tao::pegtl::tracking_mode::IMMEDIATE, tao::pegtl::ascii::eol::lf_crlf> &, States=<Actions::CFG::ActiveState &>]” at line 1055 of
“/usr/local/usrapps/iabolotn/try_10_paraview/paraview/VTK/IO/MotionFX/vtkMotionFXCFGReader.cxx”
compilation aborted for /usr/local/usrapps/iabolotn/try_10_paraview/paraview/VTK/IO/MotionFX/vtkMotionFXCFGReader.cxx (code 2)

As you can tell I am very inexperienced with compiling/building and would greatly appreciate any help you can offer. I am somewhat experienced in shell scripting, and so should be able to provide detailed logs of inputs/ outputs at specific points if needed.

Regards,

Naveen

2

Hi,
I have recently built Paraview from scratch using Paraview superbuild on a Cray hpc. I used the following files for that. Maybe you can edit them and have it working for your system. For me I just run ./config.sh <arguments> and paraview gets built and installed.
Although I faced a few errors which I was able to manually fix. Here’s more about them:
https://gitlab.kitware.com/paraview/paraview-superbuild/-/issues/157

config.sh (3.1 KB) paraview-cray-serc.cmake (3.5 KB)

3

Hi Alankar,

Thanks so much for the script and the response! I’ll be able to at least learn more from them.

I was a bit confused about how to handle the superbuilds too - do you pull both paraview and superbuild from git (as I did), and then reference the paraview version you pulled from git?

cmake -GNinja …/paraview

Or just run cmake (or I guess, your script) in the superbuild directory? I tried that first, but I guess if I do it again with what I learn from your process, it may work.

4

Hi Naveen,

I pull only Superbuild from Git. That’s all I need. Everything else is downloaded separately on its own depending on the cmake file I have provided (including Paraview) during the build process. To install a specific version of Paraview make sure you checkout for that particular version in superbuild git repo.

@npillai2 Please let me know if you face any problem.
PS: Builds with Ninja is less buggy and you may try that instead of what I did in config.sh. I also build everything in a folder outside the superbuild directory.

5

Welcome to the ParaView community, Naveen!
Two thoughts:

  • There has been a great deal of work on the Superbuild the last few years. Could you try to build 5.8.0? It will very likely give you less trouble.
  • Why are you building ParaView for a remote cluster? So you can run client/ remote server, i.e., a parallel back end? If so, pvserver (that will run on the cluster) doesn’t need Qt.

Alan

6

Hi Alan,

I appreciate the your input. My choices for Paraview versions were somewhat arbitrary - 5.2.0 was the last known version that works with the plugin my group needs (and intends to use). and 5.6.0 is the version available on the machine (a simple untar from a .gz file online, which I also have no problem doing, but we need to make our plugin with the same version, and so that’s why I’m building from scratch). I think the machine uses mesa for rendering, which I’ve recently learned is incompatible with Qt, so most of my efforts wouldn’t have worked out anyway. Unfortunately building paraview purely for the pvserver (and no GUI) will not work, as IT for the HPC claim it requires more work on their end and they will therefore not support it. Using the pvserver is actually what I (and my group) are most familiar with, and so if there’s a work then I’m all for it.

My understanding of how the superbuild works is limited because I don’t know where to start - it sounds like the readme says that after these commands I’m set to go,

$ cd paraview-superbuild
$ git fetch origin # ensure you have the latest state from the main repo
$ git checkout v5.8.0 # replace `v5.8.0` with tag name of your choice
$ git submodule update

But I can’t use make immediately, and so I assume I need to do that elsewhere (e.g. in the superbuild or projects directory, after changing some cmake files somewhere (are they all in the projects directory?) I know this is probably a simple question, but I’m pretty worn out from changing one of two things about what I’m doing re-compiling and that clearly hasn’t worked out for me. I hope you can answer it for me!

By the way, I’m definitely not opposed to building 5.8.0, and will probably try that first because as long as I get a version that works with the plugin, it doesn’t matter.

Thanks,
Naveen

7

Hi Naveen,
I would recommend the following for your case.

  • Download Paraview superbuild and checkout a particular version of interest
  • Create two directories outside the superbuild folder (one for compiling and the other for putting the compiled binaries locations of which are in my script)
  • Turn OFF or ON the libraries you need by changing my script. You may look for more options in CmakeCaheLists.txt which you can add to my .cmake script if you want.
  • You can then run config.sh to start the build process
    Once you have a GUIless pvserver (data/render/both) compiled in your remote hpc, you can start it by submitting a job script and dump the output to a file. From that file you can look at the address setup ready for connection. Now you can just port forward to the ip via ssh and connect to your remote server and thus hook up your GUI enabled Paraview client on your own local system with the pvserver running on hpc.
8

@npillai2 : are you impacted by this bug : https://gitlab.kitware.com/vtk/vtk/-/issues/17872 ?

9

@mwestphal

Thanks for showing me that thread, setting Module_vtkIOMotionFX to OFF resolves that issue for me. I ran into some errors using python3

In file included from /home/npillai2/custom_software/try_10_paraview/paraview/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c(4):
/home/npillai2/custom_software/try_10_paraview/paraview/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c(164085): error: struct "_ts" has no field "exc_traceback"
      tstate->exc_traceback = *tb;
              ^

In file included from /home/npillai2/custom_software/try_10_paraview/paraview/VTK/ThirdParty/mpi4py/vtkmpi4py/src/MPI.c(4):
/home/npillai2/custom_software/try_10_paraview/paraview/VTK/ThirdParty/mpi4py/vtkmpi4py/src/mpi4py.MPI.c(164279): error #140: too many arguments in function call
          return __Pyx_CallUnboundCMethod0(&__pyx_umethod_PyDict_Type_items, d);

which were seemingly relating to these settings (which I turned off)

 PYTHON_ENABLE_MODULE_mpi4py.MPI=OFF 
 PYTHON_ENABLE_MODULE_mpi4py.dl=OFF 
 PYTHON_MODULE_mpi4py.MPI_BUILD_SHARED=OFF 
 PYTHON_MODULE_mpi4py.dl_BUILD_SHARED=OFF

ccmake for some reason would not let me turn off all 4 of those settings

 CMake Error at VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeLists.txt:181 (set_target_properties):
   set_target_properties Can not find target to add properties to: mpi4py.MPI



 CMake Error at VTK/ThirdParty/mpi4py/vtkmpi4py/CMakeLists.txt:190 (target_link_libraries):
   Cannot specify link libraries for target "mpi4py.MPI" which is not built by
   this project.

Anyway, switching to python2 (v2.7) or setting PARAVIEW_ENABLE_PYTHON to OFF both resolved the “vtkMotionFXCFGReader” issue, resulting in a successful compile and install, but I still have a problem relating to pvserver usage (as my install did not come with a paraview GUI-based client - probably because I set PARAVIEW_BUILD_QT_GUI=OFF.

10

@dutta-alankar Thanks so much for your recommendations based on what you did. Unfortunately, I am very new to this, so there’s a lot in your script that I don’t understand, so I’d much rather try building and compiling using the pieces that I do understand (also, the machine I need to install this on doesn’t have quite the same modules to load, etc, etc).

I do, however now have a GUIless pvserver (both v5.2.0 and v5.6.0) compiled in the remote HPC, presumably with the needed plugin installed as I can see it in the Plugins directory of the build. While I’ve used a pvserver before, the critical piece of information that I need is the full hostname/IP address of the machine so I can connect to it using my v5.2.0 GUI-based client on my home computer.

Actually, trying to connect to a pvserver on the HPC(using the default Paraview 5.6.0 on the HPC that was simply put there by extracting a tar.gz file from the paraview website) is one of the first things I tried to see if I needed to build a Paraview with or without a GUI. When I e-mailed the IT responsible for the HPC usage about this (the IP to connect to), I got three responses (quoted below):

“What you are saying for pvserver will not work. (The domain name is correct.)”

“Right now, pvserver should not work. It would require modifications on our side, and we will not support that at this time.”

Later correspondence got me to the point where they assigned by group space to install and maintain custom software. Then I ran into the issues that started this thread. Given all of the above, I see two paths forward.

  1. Find a way to compile a build with the GUI-based paraview (I think their rendering engine is mesa, which is incompatible with Qt). I know very little about all of that, and so would need further help with that from someone who has done it.
  2. Make the pvserver method work. My understanding from the overall conversation had with IT was that they did not want to build paraview or otherwise put effort into installing any software, and thus did not want to “support” it in that way. Their wording made it sound like I could make it work if I knew what do.

"We don’t have the server set up. I’ve only used it (actually, the VisIT version), and have never set it up. Given our limited staff resources, I cannot commit to doing that in a reasonable amount of time, but if you are willing to help I can work with you. "

The problem is, I don’t know what constitutes “setting up” a pvserver other than building it. When I try to connect to the pvserver (from my Windows machine), I get the following error

ERROR: In C:\bbd\ecd3383f\build\superbuild\paraview\src\VTK\Common\System\vtkSocket.cxx, line 476

vtkClientSocket (000002C9252038C0): Socket error in call to connect. **No connection could be made because the target machine actively refused it.**

.

ERROR: In C:\bbd\ecd3383f\build\superbuild\paraview\src\VTK\Common\System\vtkClientSocket.cxx, line 51

vtkClientSocket (000002C9252038C0): Failed to connect to server login04.hpc.ncsu.edu:11111

Warning: In C:\bbd\ecd3383f\build\superbuild\paraview\src\ParaViewCore\ClientServerCore\Core\vtkTCPNetworkAccessManager.cxx, line 396

vtkTCPNetworkAccessManager (000002C9205DD910): Connect failed. Retrying for 53.059 more seconds.

Everything looks normal on the HPC side

Waiting for client…
Connection URL: cs://login04:11111
Accepting connection(s): login04:11111

11

Untrue, you can run ParaView (GUI) with mesa.

When I try to connect to the pvserver (from my Windows machine), I get the following error

You need to ask IT which port you can use and use it.

12

I was trying to restate this from the superbuild documentation, which may have been frivolous.

“ParaView supports multiple rendering engines including egl , mesa , osmesa , and qt5 . All of these are incompatible with each other.”

A clearer message would have been that I’ve made some attempts at trying to build a GUI-based paraview with Qt5, but it appears that that method would be in vain even if it succeeded. The question with that is how do I choose mesa as the renderer when building paraview (I didn’t see an option in ccmake, but that was a minefield - I think I saw some for osmesa, which sounds different).

Is it that simple? If so, I’ll send over an e-mail. Their responses sounded like it required more work for them than that.

13

See here for the wording issues on the superbuild rendering engines :
https://gitlab.kitware.com/paraview/paraview-superbuild/-/merge_requests/690

Their responses sounded like it required more work for them than that.

I think that is it but I can be mistaken, I’m not a network specialist.

14

So it sounds like both Qt and mesa can work together - does that mean a successful build using qt5 will work with mesa, or that building both won’t cause conflicts?

I’ve asked IT about a port for the pvserver, as that seems to be the best case scenario for me (since my earlier tests using Qt 5.9.0 failed)

15

qt5 will work with mesa

Yes it should build (and it does on our buildbots)

": internal error: 101004_111
compilation aborted for /usr/local/usrapps/iabolotn/paraview_5.2.0/paraview/Qt/Components/pqPresetToPixmap.cxx (code 4)

Which compiler are you using ? These kind of errors are usually not linked to the code itself.

16

The cmake version is 3.16.3
Everything else is part of intel 2017

The C compiler identification is Intel 17.0.1.20161005
The CXX compiler identification is Intel 17.0.1.20161005
The Fortran compiler identification is Intel 17.0.1.20161005

17

Your issue is probably linked to the usage of ICC. We do have a buildbot using icc, but without Qt.

@ben.boeckel

18

@npillai2 If you wish to use CPUs for rendering on the hpc side, you can choose OsMesa (Offscreen Mesa) as your rendering engine. You can use ENABLE_osmesa ON

If you have got the following from the pvserver on your hpc side (pvserver running on the worker nodes),

 Waiting for client…
Connection URL: cs://login04:11111
Accepting connection(s): login04:11111

Then if your client is on the same LAN (or connected to the same LAN through VPN), on your client machine you can run Paraview with the same version as your pvserver and also run the following command to port forward:
ssh <your hpc address> -L 11111:login04:11111
On your client side, open up Paraview and setup the following : image

Let me know if that helps

19

@dutta-alankar

The machine is “hpc.ncsu.edu”, and typically I login to npillai2@login.hpc.ncsu.edu, so I tried this command

ssh -L hpc.ncsu.edu 11111:login04:11111

but got this result

Bad local forwarding specification ‘hpc.ncsu.edu’

I also tried running the pvserver from a compute node (via an interactive session), but got this result

[23:55] npillai2@n2e2-4:bin $ /usr/local/usrapps/iabolotn/ParaView_Install/bin/pvserver
/usr/local/usrapps/iabolotn/ParaView_Install/lib/pvserver: error while loading shared libraries: libOpenGL.so.0: cannot open shared object file: No such file or directory

So it seems like the way it’s compiled the pvserver will only work on the login nodes

20

I think on your client machine, you need to do the following
ssh npillai2@login.hpc.ncsu.edu -L 11111:login04:11111
Then connect your Paraview to pvserver as I described earlier

Ideally you would like to run on a compute node, in interactive/non-interactive mode (via mpiexec for example). But before that you need to put the lib directory (in Paraview install) on your LD_LIBRARY_PATH using export LD_LIBRARY_PATH="<paraview install location>/lib:$LD_LIBRARY_PATH" if your Paraview is not on a standard location. However, in your case it is the OpenGL libraries that are missing which is typical for compute nodes as they lack windowed rendering feature.

For pvserver to work on compute nodes, maybe you would require windowless CPU rendering and hence you can choose OsMesa (Offscreen Mesa)

Please let me know if you can establish a connection.