Remote/headless pvserver + IndeX plugin - crash

I have access to a headless box with the following GPU

# nvidia-smi
Sun Sep 11 23:36:32 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-PCI...  Off  | 00000000:04:00.0 Off |                    0 |
| N/A   31C    P0    35W / 250W |      0MiB / 40960MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I read from this thread that the minimum requirements are CUDA 10.2+ and nvidia-driver 440.33+

Hence I believe the hardware meets the requirements

I am running Paraview 5.10.1 EGL on the headless box

I am connecting to the remote headless box using Paraview 5.10.1 MPI on a Ubuntu 18.04 box

When I try to load the IndeX plugin on the remote headless box, I get the following errors

I am able to perform remote rendering successfully with this setup, I only started looking into Nvidia IndeX today.

-----------------------------------------------------
  By loading the 'pvNVIDIAIndeX' plugin you have accepted the EULA shipped with it.
  If that is not acceptable, please restart the application without loading 
  the 'pvNVIDIAIndeX' plugin.
-----------------------------------------------------
( 654.606s) [paraview        ]         vtkSIProxy.cxx:422    ERR| vtkSIProxy (0x24041e0): Failed to create 'vtknvindex_global_settings'. This typically means that ParaView does not know about the request class to create an instance of if. Ensure that it has been correctly wrapped using the client-server wrappers and the wrapping has been initialized. Note class names are case-sensitive. Check for typos. Aborting for debugging purposes.

Loguru caught a signal: SIGABRT
Stack trace:
72            0x40802a /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/paraview-real() [0x40802a]
71      0x7f537a93cc87 __libc_start_main + 231
70            0x407c8a /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/paraview-real() [0x407c8a]
69      0x7f5377d5f120 QCoreApplication::exec() + 128
68      0x7f5377d5662a QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 298
67      0x7f53394dfe13 /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/plugins/platforms/../../lib/libQt5XcbQpa.so.5(+0x5ee13) [0x7f53394dfe13]
66      0x7f53783e90db QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 187
65      0x7f537840db35 QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*) + 261
64      0x7f537840c8ad QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) + 1837
63      0x7f5377d57dd8 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 280
62      0x7f5379d3d260 QApplication::notify(QObject*, QEvent*) + 704
61      0x7f5379d3618c QApplicationPrivate::notify_helper(QObject*, QEvent*) + 156
60      0x7f5379d908fb /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x1b68fb) [0x7f5379d908fb]
59      0x7f5379d8e467 /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x1b4467) [0x7f5379d8e467]
58      0x7f5379d3c7cc QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool) + 460
57      0x7f5377d57dd8 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 280
56      0x7f5379d3ddc8 QApplication::notify(QObject*, QEvent*) + 3624
55      0x7f5379d3618c QApplicationPrivate::notify_helper(QObject*, QEvent*) + 156
54      0x7f5379ea569b QMenu::event(QEvent*) + 171
53      0x7f5379d73c28 QWidget::event(QEvent*) + 424
52      0x7f5379ea3413 QMenu::mouseReleaseEvent(QMouseEvent*) + 403
51      0x7f5379ea257f /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x2c857f) [0x7f5379ea257f]
50      0x7f5379e9b3fc /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x2c13fc) [0x7f5379e9b3fc]
49      0x7f5379d323bc QAction::activate(QAction::ActionEvent) + 204
48      0x7f5379d2ffb2 QAction::triggered(bool) + 50
47      0x7f5377d861ea QMetaObject::activate(QObject*, int, int, void**) + 1850
46      0x7f537a5937f0 pqManagePluginsReaction::managePlugins() + 240
45      0x7f5379f14c57 QDialog::exec() + 471
44      0x7f5377d5662a QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) + 298
43      0x7f53394dfe13 /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/plugins/platforms/../../lib/libQt5XcbQpa.so.5(+0x5ee13) [0x7f53394dfe13]
42      0x7f53783e90db QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) + 187
41      0x7f537840db35 QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*) + 261
40      0x7f537840c8ad QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) + 1837
39      0x7f5377d57dd8 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 280
38      0x7f5379d3d260 QApplication::notify(QObject*, QEvent*) + 704
37      0x7f5379d3618c QApplicationPrivate::notify_helper(QObject*, QEvent*) + 156
36      0x7f5379d908fb /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x1b68fb) [0x7f5379d908fb]
35      0x7f5379d8dd86 /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x1b3d86) [0x7f5379d8dd86]
34      0x7f5379d3c7cc QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool) + 460
33      0x7f5377d57dd8 QCoreApplication::notifyInternal2(QObject*, QEvent*) + 280
32      0x7f5379d3ddc8 QApplication::notify(QObject*, QEvent*) + 3624
31      0x7f5379d3618c QApplicationPrivate::notify_helper(QObject*, QEvent*) + 156
30      0x7f5379d73c28 QWidget::event(QEvent*) + 424
29      0x7f5379e19945 QAbstractButton::mouseReleaseEvent(QMouseEvent*) + 213
28      0x7f5379e1977a /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x23f77a) [0x7f5379e1977a]
27      0x7f5379e1845a /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libQt5Widgets.so.5(+0x23e45a) [0x7f5379e1845a]
26      0x7f5379e18272 QAbstractButton::clicked(bool) + 50
25      0x7f5377d861ea QMetaObject::activate(QObject*, int, int, void**) + 1850
24      0x7f53795f95c7 pqPluginDialog::onLoadSelectedRemotePlugin() + 55
23      0x7f53795f94c1 pqPluginDialog::loadSelectedPlugins(QList<QTreeWidgetItem*>, pqServer*, bool) + 337
22      0x7f53795f6c84 pqPluginDialog::loadPlugin(pqServer*, QString const&, bool) + 68
21      0x7f5378f1a42b pqPluginManager::loadExtension(pqServer*, QString const&, QString*, bool) + 347
20      0x7f5376e8bd0f vtkSMPluginManager::LoadRemotePlugin(char const*, vtkSMSession*) + 479
19      0x7f53700448f9 /data2/nyue/systems/ParaView/ParaView-5.10.1-MPI-Linux-Python3.9-x86_64/bin/../lib/libvtkCommonCore-pv5.10.so.1(+0x5b68f9) [0x7f53700448f9]
18      0x7f5378f1befb vtkObject::vtkClassMemberCallback<pqPluginManager>::operator()(vtkObject*, unsigned long, void*) + 59
17      0x7f5377d861ea QMetaObject::activate(QObject*, int, int, void**) + 1850
16      0x7f537a5b0935 pqPluginSettingsBehavior::updateSettings() + 53
15      0x7f5376e83cf2 vtkSMParaViewPipelineController::UpdateSettingsProxies(vtkSMSession*) + 546
14      0x7f536bb65400 vtkSMParaViewPipelineControllerWithRendering::PostInitializeProxy(vtkSMProxy*) + 48
13      0x7f5376e88ac3 vtkSMParaViewPipelineController::PostInitializeProxy(vtkSMProxy*) + 163
12      0x7f5376eaa0ad vtkSMProxy::UpdateVTKObjects() + 29
11      0x7f5376efdaf2 vtkSMSettingsProxy::CreateVTKObjects() + 34
10      0x7f5376eaa843 vtkSMProxy::CreateVTKObjects() + 1091
9       0x7f5376ee03eb vtkSMSessionClient::PushState(paraview_protobuf::Message*) + 379
8       0x7f5376dc9b95 vtkPVSessionBase::PushState(paraview_protobuf::Message*) + 37
7       0x7f5376dce419 vtkPVSessionCore::PushState(paraview_protobuf::Message*) + 73
6       0x7f5376dcdc9c vtkPVSessionCore::PushStateInternal(paraview_protobuf::Message*) + 700
5       0x7f5376def754 vtkSIProxy::Push(paraview_protobuf::Message*) + 36
4       0x7f5376dee97c vtkSIProxy::InitializeAndCreateVTKObjects(paraview_protobuf::Message*) + 6604
3       0x7f5376defd0b vtkSIProxy::CreateVTKObjects() + 763
2       0x7f537a95b7f1 abort + 321
1       0x7f537a959e87 gsignal + 199
0       0x7f537a959f10 /lib/x86_64-linux-gnu/libc.so.6(+0x3ef10) [0x7f537a959f10]
( 654.610s) [paraview        ]                       :0     FATL| Signal: SIGABRT
error: exception occurred: Subprocess aborted

I just tried this exact setup here, with a GeForce RTX 3060, works perfectly, rendering of a wavelet, 200x200x200.

[glow@frollo ~/dev/paraview/others/ParaView-5.10.1-egl-MPI-Linux-Python3.9-x86_64]$ nvidia-smi 
Mon Sep 12 13:19:55 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   50C    P8    14W /  N/A |    892MiB /  6144MiB |     18%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1220      G   /usr/lib/Xorg                      69MiB |
|    0   N/A  N/A     12126    C+G   ...-x86_64/bin/pvserver-real      819MiB |
+-----------------------------------------------------------------------------+
[glow@frollo ~/dev/paraview/others/ParaView-5.10.1-egl-MPI-Linux-Python3.9-x86_64]$ ./bin/pvserver
Waiting for client...
Connection URL: cs://frollo:11111
Accepting connection(s): frollo:11111
Client connected.
-----------------------------------------------------
  By loading the 'pvNVIDIAIndeX' plugin you have accepted the EULA shipped with it.
  If that is not acceptable, please restart the application without loading 
  the 'pvNVIDIAIndeX' plugin.
-----------------------------------------------------
nvindex:   0.0   PVPLN  main info : NVIDIA IndeX ParaView plugin 5.10 using NVIDIA IndeX library 2.3 (build 348900.100.964, 26 Aug 2021, linux-x86-64-gcc7).
nvindex:   0.0   INDEX  main info : NVIDIA IndeX 2.3 (build 348900.100.964, 26 Aug 2021, linux-x86-64-gcc7) is starting up ...
nvindex:   0.0   INDEX  main info : Using default NVIDIA IndeX license.
nvindex:   0.0   INDEX  main info : Authenticating DiCE library with vendor key 'NVIDIA IndeX License for Paraview IndeX:PV:Free:v1 - 20210823 (oem:retail_cloud.20230831)'
nvindex:   0.0   INDEX  main info : This NVIDIA IndeX license key will expire on 2023-08-31.
nvindex:   0.0   INDEX  main info : This free version of NVIDIA IndeX enables the compute power of a single GPU for scientific visualization.
nvindex:   0.0   INDEX  main info : Starting the DiCE library (DiCE 2021, build 348900.100.964, 26 Aug 2021, linux-x86-64) ...
nvindex:   0.1   CUDA   rend info : Found 1 CUDA device.
nvindex:   0.1   CLUSTR net  info : Networking is switched off.
nvindex:   1.0   INDEX  main info : NVIDIA driver version: NVIDIA UNIX x86_64 Kernel Module  515.65.01  Wed Jul 20 14:00:58 UTC 2022
nvindex:   1.0   INDEX  main info : CUDA runtime version: 10.2 / CUDA driver version: 11.7
nvindex:   1.0   INDEX  main info : CUDA device 0: NVIDIA GeForce RTX 3060 Laptop GPU, Compute 8.6, 6 GB.
nvindex:   1.0   INDEX  main info : NVIDIA IndeX service started successfully.
nvindex:   1.0   PVPLN  main info : Bounding box [0 0 0; 201 201 201] is available in local memory: pv_nvindex_shm_instance_1_rank_0_timestep_0_glow.
nvindex:   1.0   PVPLN  main info : -----------------------------------------------
nvindex:   1.0   PVPLN  main info : Cluster details for NVIDIA IndeX Plugin
nvindex:   1.0   PVPLN  main info : -----------------------------------------------
nvindex:   1.0   PVPLN  main info : Total number of MPI ranks: 1
nvindex:   1.0   PVPLN  main info : Total number of hosts: 1
nvindex:   1.0   PVPLN  main info : ------------------
nvindex:   1.0   PVPLN  main info : Data properties: 
nvindex:   1.0   PVPLN  main info : Scalar type: float
nvindex:   1.0   PVPLN  main info : Volume bbox: [-100 -100 -100; 100 100 100]
nvindex:   1.0   PVPLN  main info : Volume size: [200 200 200]
nvindex:   1.0   PVPLN  main info : Data values: [33.9511 286.341]
nvindex:   1.0   PVPLN  main info : Time series: No
nvindex:   1.0   PVPLN  main info : ------------------
nvindex:   1.0   PVPLN  main info : Host properties: 
nvindex:   1.0   PVPLN  main info : Host name: frollo, hostid: 1
nvindex:   1.0   PVPLN  main info : Rank ids: 0 
nvindex:   1.0   PVPLN  main info : Shared memory pieces [shmname, shmbbox]:
nvindex:   1.0   PVPLN  main info : Time step: 0 pv_nvindex_shm_instance_1_rank_0_timestep_0_glow:[0 0 0; 201 201 201]
nvindex:   1.0   PVPLN  main info : -----------------------------------------------
nvindex:   1.0   INDEX  main info : Initializing CUDA run-time compiler for device capability 8.6
nvindex:   1.0   INDEX  main info : CUDA run-time compiler will use compute capability 7.5
nvindex:   1.0   INDEX  main info : Building a spatial subdivision that spans the user-defined domain (bbox): [0 0 0; 201 201 201].
nvindex:   1.0   INDEX  main info : The size of the spatial subdivision has been increased; the new bounding box is [0 0 0; 201 201 201]
nvindex:   1.0   INDEX  main info : Reserving compositing resources assigned for rendering issued from host 1.
nvindex:   1.4   PVPLN  main info : Importing volume data from local memory (pv_nvindex_shm_instance_1_rank_0_timestep_0_glow) on rank 0, data bbox [0 0 0; 201 201 201], importer bbox [-2 -2 -2; 203 203 203], border 0/2.
Exiting...

Hmmm … I tried again and now it works. Exact same setup.

Thanks for confirming @mwestphal