pvserver seg fault

I am getting the following dump when trying to run pvserver 5.9.1 in reverse-connection mode.
On the client side it appears to read the file (openfoam reader) and then if fails to render on screen getting the following.
I am total lost as to where/how to debug this problem.
Any help is appreciated.
Thanks


Process id 887493 Caught SIGSEGV at 0x68 address not mapped to object
Program Stack:
WARNING: The stack trace will not use advanced capabilities because this is a release build.
0x147f13092410 : ??? [(???) ???:-1]
0x147efbde0c23 : ??? [(???) ???:-1]
0x147efbdd6aea : OSMesaCreateContextAttribs [(libOSMesa.so.8) ???:-1]
0x147efd302623 : vtkOSOpenGLRenderWindow::CreateOffScreenWindow(int, int) [(libvtkRenderingOpenGL2-pv5.9.so.1) ???:-1]
0x147efd28f0ca : vtkOpenGLRenderWindow::Start() [(libvtkRenderingOpenGL2-pv5.9.so.1) ???:-1]
0x147ef8dba275 : vtkRenderWindow::Render() [(libvtkRenderingCore-pv5.9.so.1) ???:-1]
0x147efd291172 : vtkOpenGLRenderWindow::Render() [(libvtkRenderingOpenGL2-pv5.9.so.1) ???:-1]
0x147f0c9c7d32 : vtkPVProcessWindow::PrepareForRendering() [(libvtkRemotingViews-pv5.9.so.1) ???:-1]
0x147f0c9efa25 : ??? [(???) ???:-1]
0x147f0c9daefe : vtkPVRenderView::Render(bool, bool) [(libvtkRemotingViews-pv5.9.so.1) ???:-1]
0x147f0c9d0cd1 : vtkPVRenderView::StillRender() [(libvtkRemotingViews-pv5.9.so.1) ???:-1]
0x147f11dd0138 : vtkPVRenderViewCommand(vtkClientServerInterpreter*, vtkObjectBase*, char const*, vtkClientServerStream const&, vtkClientServerStream&, voi$
0x147f1082d8c2 : vtkClientServerInterpreter::CallCommandFunction(char const*, vtkObjectBase*, char const*, vtkClientServerStream const&, vtkClientServerStr$
0x147f1082de25 : vtkClientServerInterpreter::ProcessCommandInvoke(vtkClientServerStream const&, int) [(libvtkRemotingClientServerStream-pv5.9.so.1) ???:-1]
0x147f1082e2ce : vtkClientServerInterpreter::ProcessOneMessage(vtkClientServerStream const&, int) [(libvtkRemotingClientServerStream-pv5.9.so.1) ???:-1]
0x147f1082e38d : vtkClientServerInterpreter::ProcessStream(vtkClientServerStream const&) [(libvtkRemotingClientServerStream-pv5.9.so.1) ???:-1]
0x147f114eb892 : vtkPVSessionCore::ExecuteStreamInternal(vtkClientServerStream const&, bool) [(libvtkRemotingServerManager-pv5.9.so.1) ???:-1]
0x147f114ebac2 : vtkPVSessionCore::ExecuteStreamSatelliteCallback() [(libvtkRemotingServerManager-pv5.9.so.1) ???:-1]
0x147f0fc64c50 : vtkMultiProcessController::ProcessRMI(int, void*, int, int) [(libvtkParallelCore-pv5.9.so.1) ???:-1]
0x147f0fc64e7f : vtkMultiProcessController::BroadcastProcessRMIs(int, int) [(libvtkParallelCore-pv5.9.so.1) ???:-1]
0x147f0fc65195 : vtkMultiProcessController::ProcessRMIs(int, int) [(libvtkParallelCore-pv5.9.so.1) ???:-1]
0x4016ee : ??? [(???) ???:-1]
0x147f1307e4a3 : __libc_start_main [(libc.so.6) ???:-1]
0x40175e : ??? [(???) ???:-1]

You need to share data and steps to reproduce.

on the server side (cluster) did a
mpirun -n 4 -N 4 -x KNOB_MAX_WORKER_THREADS=6 -x GALLIUM_DRIVER=swr -x LP_NUM_THREADS=6
pvserver -rc -ch=192.168.11.21 -sp=64254 --enable-bt --force-offscreen-rendering

and on the client side (windows) I connected just fine to the session, then navigated to the openfoam case specified case type: decomposed case grid and unchecked skip zero time. Clicked apply saw progress bar for reading openfoam complete successfully and the after the second progress bar (geometry representation with faces) it pauses then no image is rendered and I get a dialog on the client “The server side has disconnected. The application will now quit since it may be in an unrecoverable state.” At this point the submission has crashed with the dump previously provided.
I am using version 5.9.1 on Rocky 8.4, openmpi4.1.1, osemesa21.1.2, llvm12.0.1, gcc8.4.1 and with 32 core compute nodes (no gpu).
I can tell you that in forward connection mode everything works well it is only in reverse connection that we have a problem.
I hope this is enough detail.

Looks like a problem with your data segfaulting pvserver.

hmmm how did you come to that conclusion when the same data works fine in forward connection mode?

I missed that info. Good point. Can you reproduce with a non distributed server ?

I took a different approach in troubleshooting and confirmed that the prebuilt binaries worked in reverse mode but rather badly in performance. This leads me down the rabbit hole of it being a compile problem somewhere in the paraview chain, i.e. VTK, QT, LLVM, OSMESA and Paraview itself.
For me to explore this rabbit hole I have been trying to find the proper build options for the versions I stated but no luck yet. Do you perhaps have a list of build options which I can rebuild everything to. Keep in mind the versions of everything I stated.
Thanks