hardware accelerated headless rendering in version 6.0

jfavre · May 21, 2025, 9:08am

We have ParaView-v6.0.0-RC1 compiled on a Grace Hopper cluster with the standard cmake options -DVTK_OPENGL_HAS_EGL:BOOL=ON -DVTK_USE_X:BOOL=OFF

According to the release notes, the variable VTK_DEFAULT_OPENGL_WINDOW can be used to force a specific backend
We set it to vtkEGLRenderWindow, and expects pvbatch to report the same output which we have with version v5.13 also compiled with the same options and the same compiler. Our baseline (v5.13) reports:

Vendor: NVIDIA Corporation
Version: 4.6.0 NVIDIA 550.54.15
Renderer: NVIDIA GH200 120GB/PCIe

whereas v6.0 reports:

Vendor: Mesa/X.org
Version: 3.3 (Core Profile) Mesa 22.3.5
Renderer: softpipe

our observation is that our distribution includes two different libEGL*so version, namely, /usr/lib64/libEGL_mesa.so, /usr/lib64/libEGL.so

and v6.0 seems to pick us the mesa version, independently of our forcing it to use the NVIDIA version.

Are there tips on debugging this issue?
TIA

mwestphal · May 21, 2025, 9:37am

I think the dynamic loading must find the libEGL_mesa.so for some reason ?

What happens if you move it temporarly ?

FYI @jaswantp

jfavre · May 21, 2025, 11:43am

I cannot move it. This is a production system. I have no system rights.

mwestphal · May 21, 2025, 11:46am

does using LD_PRELOAD has an effect ?

jfavre · May 21, 2025, 11:52am

LD_PRELOAD=/usr/lib64/libEGL.so ./bin/pvbatch -m paraview.info

OpenGL Vendor : Mesa/X.org
OpenGL Version : 3.3 (Core Profile) Mesa 22.3.5
OpenGL Renderer : softpipe
OpenGL Window Backend : OSMesa

mwestphal · May 21, 2025, 11:59am

I’m out of ideas, but maybe @jaswantp will have another one.

In short we rely on gladLoaderLoadEGL to load the lib.

jaswantp · May 21, 2025, 1:11pm

Not possible, the EGL libraries that glad searches for are:

static const char *NAMES[] = {"libEGL.so.1", "libEGL.so"};

Moreover, pvbatch -m paraview.info printed “OpenGL Window Backend : OSMesa” so this is unlikely. In short, it used vtkOSOpenGLRenderWindow after realizing that vtkEGLRenderWindow failed to initialize.

@jfavre have you tried eglinfo?

mwestphal · May 21, 2025, 1:20pm

I missed that.

jfavre · May 21, 2025, 1:28pm

there is no eglinfo installed on my system

I have tried

./bin/vtkProbeOpenGLVersion-pv6.0
2025-05-21 15:26:52.888 ( 0.020s) [ FFFFA35AB370]vtkOpenGLRenderWindow.c:286 WARN| GL Message: id=1 source=SOURCE_API(0x8246) type=ERROR(0x824c) severity=HIGH(0x9146) message=GL_INVALID_ENUM in glGetIntegerv(pname=GL_MAX_TESS_GEN_LEVEL)
Class: vtkOSOpenGLRenderWindow succeeded in finding a working OpenGL

OpenGL vendor string: Mesa/X.org
OpenGL renderer string: softpipe
OpenGL version string: 3.3 (Core Profile) Mesa 22.3.5

jfavre · May 21, 2025, 1:30pm

to compare with our v5.13 version:

./bin/vtkProbeOpenGLVersion-pv5.13
Class: vtkEGLRenderWindow succeeded in finding a working OpenGL

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: NVIDIA GH200 120GB/PCIe
OpenGL version string: 4.6.0 NVIDIA 550.54.15

jaswantp · May 21, 2025, 1:36pm

Can you try setting VTK_DEFAULT_EGL_DEVICE_INDEX to 1 or 2? It might be that it needs to try different device.

what happens if you did not set VTK_DEFAULT_OPENGL_WINDOW environment variable and just ran ./bin/vtkProbeOpenGLVersion-pv6.0?

Btw, the default value for VTK_OPENGL_HAS_EGL is ON on UNIX platform.

jfavre · May 21, 2025, 1:45pm

I have 4 GPUs available, and can set VTK_DEFAULT_EGL_DEVICE_INDEX to 0,1,2 or 3 and the answer is always the same

the results is also independent of setting/unsetting VTK_DEFAULT_OPENGL_WINDOW

jaswantp · May 21, 2025, 1:52pm

Okay. It should atleast report some warnings if vtkEGLRenderWindow fails to find a device.
I expected to see atleast one of these warning messages:

from vtkEGLRenderWindow: "Getting the number of devices (graphics cards) on a system require "
“EGL_EXT_device_base, EGL_EXT_platform_device and EGL_EXT_platform_base extensions”
from vtkEGLRenderWindow: “EGL device index: 0 could not be initialized. Trying other devices…”
from vtkEGLRenderWindow: “Could not initialize a device. Exiting…”
from vtkEGLRenderWindow: "Only EGL 1.4 and greater allows OpenGL as client API. "
“See eglBindAPI for more information.”
from vtkOpenGLRenderWindow: “Failed to initialize OpenGL functions!”

I’m surprised it doesn’t report a single warning besides that GL_MAX_TESS_GEN_LEVEL. This could only mean that vtkEGLRenderWindow was not compiled at all?

What are the contents of $buildDir/VTK/Rendering/OpenGL2/vtkRenderingOpenGLConfigure.h? This file also exists in the install directory. Does it have a #define VTK_OPENGL_HAS_EGL? Is there a vtkEGLRenderWindow.cxx.o object in your build dir?

jfavre · May 21, 2025, 1:57pm

more vtkRenderingOpenGLConfigure.h
// SPDX-FileCopyrightText: Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
// SPDX-License-Identifier: BSD-3-Clause

#ifndef vtkRenderingOpenGLConfigure_h
#define vtkRenderingOpenGLConfigure_h

/* This header contains build settings for the vtkRenderingOpenGL2 module. /
/ It provides some compatibility with the vtkRenderingOpenGL module. */

#ifndef VTK_OPENGL2
#define VTK_OPENGL2
#endif

/* Whether VTK was built to support Cocoa on the Mac. /
/ #undef VTK_USE_COCOA */

/--------------------------------------------------------------------------/
/* Rendering Configuration /
/ #undef VTK_USE_X /
#define VTK_USE_OPENGL_LIBRARY
#define VTK_OPENGL_HAS_EGL
/ #undef VTK_DEFAULT_RENDER_WINDOW_OFFSCREEN /
/ #undef VTK_OPENGL_ENABLE_STREAM_ANNOTATIONS */

// for legacy codes (don’t use these in new codes)
#ifdef VTK_DEFAULT_RENDER_WINDOW_OFFSCREEN

define VTK_USE_OFFSCREEN

#endif

/* Options for GPUInfo /
/ #undef VTK_USE_DIRECTX /
/ #undef VTK_USE_CORE_GRAPHICS /
/ #undef VTK_USE_NVCONTROL */

/--------------------------------------------------------------------------/
/* Setup VTK based on platform features and configuration. */

/* OGLR /
/ OGLR is for GLX. It can be on Unix, Mac or Windows */
#if ((defined(VTK_USE_OPENGL_LIBRARY) && defined(VTK_USE_X) &&
!defined(_WIN32)) ||
(defined(VTK_USE_X) && defined(_WIN32))) &&
!defined(VTK_USE_COCOA)

define VTK_USE_OGLR

#endif

find . -name vtkEGLRenderWindow.cxx.o
./CMakeFiles/RenderingOpenGL2.dir/vtkEGLRenderWindow.cxx.o

jaswantp · May 21, 2025, 2:06pm

Okay, thanks for sharing. That confirms its part of the build.

Is this a debug build? or a release build?

If it’s release, can you apply this patch in VTK subdir and rebuild?

diff --git a/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx b/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx
index 435d7d439a..aaf3f657bb 100644
--- a/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx
+++ b/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx
@@ -644,6 +644,7 @@ vtkOpenGLRenderWindow* vtkOpenGLRenderWindow::New()
 #if defined(VTK_OPENGL_HAS_EGL)
   if ((backend == nullptr) || (std::string(backend) == "vtkEGLRenderWindow"))
   {
+    vtkLogF(INFO, "VTK_OPENGL_HAS_EGL is defined, trying gladLoaderLoadEGL");
     // Load core egl functions.
     if (!gladLoaderLoadEGL(EGL_NO_DISPLAY))
     {
@@ -652,12 +653,17 @@ vtkOpenGLRenderWindow* vtkOpenGLRenderWindow::New()
     }
     else
     {
+      vtkLogF(INFO, "gladLoaderLoadEGL succeeded");
       vtkNew<vtkEGLRenderWindow> eglRenderWindow;
       eglRenderWindow->Initialize();
       if (eglRenderWindow->Initialized)
       {
         return eglRenderWindow->NewInstance();
       }
+      else
+      {
+        vtkLogF(INFO, "eglRenderWindow->Initialize() failed, trying vtkOSOpenGLRenderWindow");
+      }
     }
   }
 #endif

jfavre · May 21, 2025, 2:59pm

Did I misunderstand you? I have a Release build, but I don’t see any other log messages. Do I need an extra env variable?

[daint][jfavre@nid006325 ParaView-v6.0.0-RC1Build-EGL]$ ./bin/pvbatch -m paraview.info
ParaView Version : 6.0.0
VTK Version : 9.4.20250513
MPI Enabled : True
MPI Rank : 0
MPI Size : 1
Python Library Path : /user-environment/linux-sles15-neoverse_v2/gcc-13.3.0/python-3.12.5-tiwaj3tabjlxhozzutmhoxawzy6pmm5s/lib/python3.12
Python Library Version : 3.12.5 (main, Jan 1 1980, 12:01:00) [GCC 13.3.0]

OpenGL Vendor : Mesa/X.org
OpenGL Version : 3.3 (Core Profile) Mesa 22.3.5
OpenGL Renderer : softpipe
OpenGL Window Backend : OSMesa
Disable Registry : False
HostName : nid006325
Client HostName : localhost

=====================================================
btw, I am not sure your paraview.info can be trusted at 100%. Here is why:
I did not enable MPI:
grep PARAVIEW_USE_MPI CMakeCache.txt
PARAVIEW_USE_MPI:BOOL=OFF

Yet, the paraview.info module reports “MPI Enabled : True” (see above)

jaswantp · May 21, 2025, 3:05pm

So the warnings should appear even on release builds. The reason I asked if it is a release build is because it’s easier to debug with print statements instead of stepping through in a debugger. Have you applied the patch I shared and rebuilt? Does it print the sentences in vtkLog from the patch file?

Very strange indeed!

jfavre · May 21, 2025, 4:35pm

yes, I have applied the patch, and cannot get any log message printed.
I am making another build in Debug mode

jaswantp · May 21, 2025, 4:38pm

Okay. In debug mode, i would be interested if you can break at vtkOpenGLRenderWindow::New if it ever gets there that is.

jfavre · May 21, 2025, 5:16pm

I am not getting there. I put a breakpoint in vtkOpenGLMessageHandler and this is how I got there:

199 std::string messageType;
Missing separate debuginfos, use: zypper install Mesa-libglapi0-debuginfo-22.3.5-150500.75.2.aarch64 libOSMesa8-debuginfo-22.3.5-150500.75.2.aarch64
(gdb) where
#0 vtkOpenGLMessageHandler (source=33350, type=33356, id=1, severity=37190,
message=0xffffffffb3e8 “GL_INVALID_ENUM in glGetIntegerv(pname=GL_MAX_TESS_GEN_LEVEL)”)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx:199
#1 0x0000ffffd32fcc1c in ?? () from /usr/lib64/libOSMesa.so.8
#2 0x0000ffffd348199c in ?? () from /usr/lib64/libOSMesa.so.8
#3 0x0000ffffd3483a9c in ?? () from /usr/lib64/libOSMesa.so.8
#4 0x0000ffffe9c4e4a8 in vtkOpenGLState::Initialize (this=0xa3ae50)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLState.cxx:1602
#5 0x0000ffffe9c0fffc in vtkOpenGLRenderWindow::OpenGLInitState (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx:837
#6 0x0000ffffe9c0ffb0 in vtkOpenGLRenderWindow::OpenGLInit (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx:825
#7 0x0000ffffe9c92f58 in vtkOSOpenGLRenderWindow::CreateOffScreenWindow (this=0xa3a270,
width=300, height=300)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOSOpenGLRenderWindow.cxx:322
#8 0x0000ffffe9c931b8 in vtkOSOpenGLRenderWindow::Initialize (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOSOpenGLRenderWindow.cxx:393
#9 0x0000ffffe9c14ef0 in vtkOpenGLRenderWindow::Start (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx:1913
#10 0x0000ffffe7c784d4 in vtkRenderWindow::Render (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/Core/vtkRenderWindow.cxx:450
#11 0x0000ffffe9c18ae4 in vtkOpenGLRenderWindow::Render (this=0xa3a270)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/VTK/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx:3064
#12 0x0000fffff01ea540 in vtkPVOpenGLInformation::CopyFromObject (this=0xa3a0e0, obj=0x0)
at /capstor/scratch/cscs/jfavre/ParaView/ParaView-v6.0.0-RC1/Remoting/Views/vtkPVOpenGLInformation.cxx:55

is discourse still the best way to communicate about this issue? feel free to engage in discussion with me in another channel.