failure in line 157 retVal = func(*newArgs)

After compilation of Paraview 5.10.0 (using superbuild) on a Cray XC40 with GNU (11.2) compilers and with osmesa build (batch mode only). Simple Paraview scripts run fine. Trying to display a Nek500 input file of 17 GB an error occurred. My experience in debugging and running Paraview in batch mode is very limited and I tried to find a cause of the problem. Tracking the error with breakpoint() showed that the code fails in:

> /lustre/scratch/x_thorbej/Paraview/test_pv/<module>()
-> renderView1.Update()
(Pdb) s
> /sw/xc40cle7up03/paraview/5.10.0/cle73_gnu1120/lib/python3.9/site-packages/paraview/
.... left out next steps ....
> /sw/xc40cle7up03/paraview/5.10.0/cle73_gnu1120/lib/python3.9/site-packages/paraview/
-> func = getattr(self.SMProxy, self.__LastAttrName)
(Pdb) n
> /sw/xc40cle7up03/paraview/5.10.0/cle73_gnu1120/lib/python3.9/site-packages/paraview/
-> retVal = func(*newArgs)
(Pdb) s
srun: error: nid00754: task 0: Killed
srun: Terminating job step 24923364.0
srun: Force Terminated job step 24923364.0

At the moment I do not know how to continue to track down and solve the error. The Paraview commands are run in a slurm batch script with the following settings:


srun -N1 -n1 pvbatch

The contains the commands:

#### import the simple module from the paraview
from paraview.simple import *
#### disable automatic camera reset on 'Show'

# create a new 'VisItNek5000Reader'
g0p1_300nek5000 = VisItNek5000Reader(registrationName='G0p1_400.nek5000', FileName='G0p1_400.nek5000')
g0p1_300nek5000.Meshes = ['mesh']
g0p1_300nek5000.PointArrays = []

# get animation scene
animationScene1 = GetAnimationScene()

# update animation scene based on data timesteps

# Properties modified on g0p1_300nek5000
g0p1_300nek5000.PointArrays = ['temperature']

# get active view
renderView1 = GetActiveViewOrCreate('RenderView')

# show data in view
g0p1_300nek5000Display = Show(g0p1_300nek5000, renderView1, 'UnstructuredGridRepresentation')

#print('Before g0p1_300nek5000Display.Representation')
# trace defaults for the display properties.
g0p1_300nek5000Display.Representation = 'Surface'
g0p1_300nek5000Display.ColorArrayName = [None, '']
g0p1_300nek5000Display.SelectTCoordArray = 'None'
g0p1_300nek5000Display.SelectNormalArray = 'None'
g0p1_300nek5000Display.SelectTangentArray = 'None'
g0p1_300nek5000Display.OSPRayScaleArray = 'temperature'
g0p1_300nek5000Display.OSPRayScaleFunction = 'PiecewiseFunction'
g0p1_300nek5000Display.SelectOrientationVectors = 'None'
g0p1_300nek5000Display.ScaleFactor = 0.1
g0p1_300nek5000Display.SelectScaleArray = 'None'
g0p1_300nek5000Display.GlyphType = 'Arrow'
g0p1_300nek5000Display.GlyphTableIndexArray = 'None'
g0p1_300nek5000Display.GaussianRadius = 0.005
g0p1_300nek5000Display.SetScaleArray = ['POINTS', 'temperature']
g0p1_300nek5000Display.ScaleTransferFunction = 'PiecewiseFunction'
g0p1_300nek5000Display.OpacityArray = ['POINTS', 'temperature']
g0p1_300nek5000Display.OpacityTransferFunction = 'PiecewiseFunction'
g0p1_300nek5000Display.DataAxesGrid = 'GridAxesRepresentation'
g0p1_300nek5000Display.PolarAxes = 'PolarAxesRepresentation'
g0p1_300nek5000Display.ScalarOpacityUnitDistance = 0.0058355046803493975
g0p1_300nek5000Display.OpacityArrayName = ['POINTS', 'temperature']
# reset view to fit data

# get the material library
materialLibrary1 = GetMaterialLibrary()
# update the view to ensure updated data information

… The script continues with more commands…

It could be that in the file commands are missing. in the wrong order, and are the cause of the failure.
When the renderView1.Update() command and the renderView1.ResetCamera(False) are commented out the script continues without an error and produces a png file (not shown in script snippet). I also tried Paraview 5.9.1 and 5.10.1 and they give the same error.
The same script also works fine when smaller Nek500 snapshots (~4 GB) are displayed.
Any help or advice is highly appreciated.

to close the loop on this issue, I advised the user to compile a parallel-aware NEK5000 reader which I have developed, See GitHub - jfavre/ParaViewNek5000Plugin for the source code. Given an allocation with N MPI tasks, the reader will distribute the mesh (2D and 3D are supported) across compute nodes.
The user reported that replacing the serial reader from the VisIt code base with this new reader solved the problem.