error with mesa library

Hi,

I am getting error when i try to use my application (along with co-processing) with mesa library (the same code is working without any problem with NVIDIA EGL installation). I tried both Python 2 and Python 3 with PV 5.6.1. In Python 2 the code gives error related with PyUnicodeUCS2_DecodeUTF8 and Python 3 give error related with MPI_Send. I also tried to compile ParaView in debug mode but I could not get any further information. Any suggestion?

Python 2 Error:

Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "/glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa_system_py/lib/python2.7/site-packages/paraview/servermanager.py", line 53, in <module>
    from paraview import vtk
  File "/glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa_system_py/lib/python2.7/site-packages/paraview/vtk.py", line 12, in <module>
    all_m = importlib.import_module('paraview.pv-vtk-all')
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa_system_py/lib/python2.7/site-packages/paraview/pv-vtk-all.py", line 7, in <module>
    from vtkmodules.vtkCommonCore import *
  File "/glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa_system_py/lib/python2.7/site-packages/vtkmodules/vtkCommonCore.py", line 9, in <module>
    from vtkCommonCorePython import *
ImportError: /glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa_system_py/lib/libvtkCommonCorePython27D-pv5.6.so.1: undefined symbol: PyUnicodeUCS2_DecodeUTF8

Python 3 Error:

Abort(738846724) on node 278 (rank 278 in comm 0): Fatal error in PMPI_Send: Invalid tag, error stack:
PMPI_Send(159): MPI_Send(buf=0x7fffc3873320, count=1, MPI_INT, dest=138, tag=1278392, comm=0xc400000a) failed
PMPI_Send(101): Invalid tag, value is 1278392
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
regesm.x           00000000032C1B64  for__signal_handl     Unknown  Unknown
libpthread.so.0    00002B3FA4CA9B00  Unknown               Unknown  Unknown
libc.so.6          00002B3FC8C836B3  __open_catalog        Unknown  Unknown
....
Abort(738846724) on node 278 (rank 278 in comm 0): Fatal error in PMPI_Send: Invalid tag, error stack:
PMPI_Send(159): MPI_Send(buf=0x7fffc3873320, count=1, MPI_INT, dest=138, tag=1278392, comm=0xc400000a) failed
PMPI_Send(101): Invalid tag, value is 1278392
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
regesm.x           00000000032C1B64  for__signal_handl     Unknown  Unknown
libpthread.so.0    00002B3FA4CA9B00  Unknown               Unknown  Unknown
libc.so.6          00002B3FC8C836B3  __open_catalog        Unknown  Unknown 

regesm.x           00000000032C1B64  for__signal_handl     Unknown  Unknown
libpthread.so.0    00002AC724651B00  Unknown               Unknown  Unknown
libmpi.so.12.0.0   00002AC746B9F801  MPII_Datatype_pri         286  type_debug.c
libmpi.so.12.0.0   00002AC745CCF09E  GetDTypeString           1404  errutil.c
libmpi.so.12.0.0   00002AC745CD011D  vsnprintf_mpi            1611  errutil.c
libmpi.so.12.0.0   00002AC745CCD98B  MPIR_Err_create_c         995  errutil.c
libmpi.so.12.0.0   00002AC745CCD390  MPIR_Err_create_c         862  errutil.c
libmpi.so.12.0.0   00002AC7468718D4  PMPI_Recv                 171  recv.c
libvtkParallelMPI  00002AC727A082A5  _ZN18vtkMPICommun     Unknown  Unknown
libvtkParallelMPI  00002AC727A03A0B  _ZN18vtkMPICommun     Unknown  Unknown
libvtkParallelCor  00002AC72A9B3984  _ZN15vtkCommunica     Unknown  Unknown
libvtkPVVTKExtens  00002AC740ED28E0  _ZN31vtkMultiProc     Unknown  Unknown
libvtkPVVTKExtens  00002AC742E64DF0  _ZN14vtkCleanArra     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAFA7D  _ZN12vtkExecutive     Unknown  Unknown
libvtkCommonExecu  00002AC72AEABAD9  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA601A  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA601  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAB705  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDCB35  _ZN32vtkStreaming     Unknown  Unknown
libvtkPVVTKExtens  00002AC742E78EDB  _ZN33vtkDataObjec     Unknown  Unknown
libvtkPVVTKExtens  00002AC742E78C57  _ZN33vtkDataObjec     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAFA7D  _ZN12vtkExecutive     Unknown  Unknown
libvtkCommonExecu  00002AC72AEABAD9  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA601A  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA601  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA59EA  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA5AC  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA59EA  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA5AC  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA59EA  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA5AC  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA59EA  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA5AC  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEA59EA  _ZN24vtkComposite     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAA5AC  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDC6CF  _ZN32vtkStreaming     Unknown  Unknown
libvtkCommonExecu  00002AC72AEAB705  _ZN23vtkDemandDri     Unknown  Unknown
libvtkCommonExecu  00002AC72AEDCB35  _ZN32vtkStreaming     Unknown  Unknown
libvtkPVServerImp  00002AC7407B07BD  _ZN16vtkSISourceP     Unknown  Unknown
libvtkPVServerMan  00002AC729B7043C  _Z23vtkSISourcePr     Unknown  Unknown
libvtkClientServe  00002AC744840778  _ZN26vtkClientSer     Unknown  Unknown
libvtkClientServe  00002AC74483F06D  _ZN26vtkClientSer     Unknown  Unknown
libvtkClientServe  00002AC74483C61C  _ZN26vtkClientSer     Unknown  Unknown
libvtkClientServe  00002AC744840569  _ZN26vtkClientSer     Unknown  Unknown
libvtkPVServerImp  00002AC740766C67  _ZN16vtkPVSession     Unknown  Unknown
libvtkPVServerImp  00002AC7407641F5  _ZN16vtkPVSession     Unknown  Unknown
libvtkPVServerImp  00002AC740761740  _ZN16vtkPVSession     Unknown  Unknown
libvtkPVServerMan  00002AC736DEA08B  _ZN15vtkSMOutputP     Unknown  Unknown
libvtkPVServerMan  00002AC736E92AF9  _ZN16vtkSMSourceP     Unknown  Unknown
libvtkPVServerMan  00002AC74017A783  _ZN44vtkSMParaVie     Unknown  Unknown
libvtkPVServerMan  00002AC740178285  _ZN44vtkSMParaVie     Unknown  Unknown
libvtkPVServerMan  00002AC759D92610  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC4EAB1  PyCFunction_Call      Unknown  Unknown
libpython3.6m.so.  00002AC73ACBE9D0  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59A9C  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59A9C  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AB9061E  PyEval_EvalCodeEx     Unknown  Unknown
libpython3.6m.so.  00002AC73ACB714B  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC4E96A  _PyCFunction_Fast     Unknown  Unknown
libpython3.6m.so.  00002AC73AC59B8C  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59A9C  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59A9C  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73ABB66B3  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59C5E  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73ABB66B3  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AC59C5E  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73ACBC232  _PyEval_EvalFrame     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9049F  Unknown               Unknown  Unknown
libpython3.6m.so.  00002AC73AB9061E  PyEval_EvalCodeEx     Unknown  Unknown
libpython3.6m.so.  00002AC73AB9064B  PyEval_EvalCode       Unknown  Unknown
libpython3.6m.so.  00002AC73ACDB22B  PyRun_StringFlags     Unknown  Unknown
libpython3.6m.so.  00002AC73ACE0D7B  PyRun_SimpleStrin     Unknown  Unknown
libvtkPythonInter  00002AC72A794577  _ZN20vtkPythonInt     Unknown  Unknown
libvtkPVPythonCat  00002AC7277F37F3  _ZN25vtkCPPythonS     Unknown  Unknown
libvtkPVCatalyst-  00002AC7275E42C2  _ZN14vtkCPProcess     Unknown  Unknown
regesm.x           00000000032A3AC4  Unknown               Unknown  Unknown
regesm.x           000000000045D028  Unknown               Unknown  Unknown
libesmf.so         00002AC71FEE2E99  _ZN5ESMCI13Method     Unknown  Unknown
libesmf.so         00002AC71FEE2DCA  _ZN5ESMCI11Method     Unknown  Unknown
libesmf.so         00002AC71FEE21E2  c_esmc_methodtabl     Unknown  Unknown
libesmf.so         00002AC7200AA302  esmf_attachmethod     Unknown  Unknown
libesmf.so         00002AC72070BCD5  nuopc_modelbase_m     Unknown  Unknown
libesmf.so         00002AC71FBEC4A2  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00002AC71FBEFA35  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00002AC71FFE5B65  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00002AC71FBED858  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00002AC7201E270D  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00002AC720387E71  esmf_gridcompmod_     Unknown  Unknown
libesmf.so         00002AC7206DC354  nuopc_driver_mp_r     Unknown  Unknown
libesmf.so         00002AC71FBEC4A2  _ZN5ESMCI6FTable1     Unknown  Unknown
libesmf.so         00002AC71FBEFA35  ESMCI_FTableCallE     Unknown  Unknown
libesmf.so         00002AC71FFE5B65  _ZN5ESMCI2VM5ente     Unknown  Unknown
libesmf.so         00002AC71FBED858  c_esmc_ftablecall     Unknown  Unknown
libesmf.so         00002AC7201E270D  esmf_compmod_mp_e     Unknown  Unknown
libesmf.so         00002AC720387E71  esmf_gridcompmod_     Unknown  Unknown
regesm.x           00000000004193FA  Unknown               Unknown  Unknown
regesm.x           0000000000418FC2  Unknown               Unknown  Unknown
libc.so.6          00002AC7486186E5  __libc_start_main     Unknown  Unknown
regesm.x           0000000000418EC9  Unknown               Unknown  Unknown

It looks to me like there may be a mismatch between the ParaView against which you built ParaView and the one that is being used at runtime. For your Python 2 build, what does

ldd bin/paraview | grep libpython

report?

And in your CMake configuration, what is the value of Python2_LIBRARY_RELEASE?

Hi,

Thanks for your help, in my case there is no ParaView binary because i
did not compile GUI on cluster. If i run same command with pvpython, i
am getting following

ldd ~/progs/paraview-5.6.1/install_mesa/bin/pvpython | grep libpython

 libpython2.7.so.1.0 => 

/glade/u/home/turuncu/progs/python-2.7.16/lib/libpython2.7.so.1.0
(0x00002b85b889a000)

which points the correct python installation. In my case there is no
Python2_LIBRARY_RELEASE variable in CMake configuration. I am using
5.6.1. Is this a problem?

Regards,

–ufuk

BTW, i also recompile Python again without --enable-unicode=ucs4 option but i am still getting following error in co-processing step.

forrtl: error (76): Abort trap signal
Image PC Routine Line Source
regesm.x 000000000360476E Unknown Unknown Unknown
libpthread.so.0 00002B7952440B00 Unknown Unknown Unknown
libc.so.6 00002B7973FD58D7 gsignal Unknown Unknown
libc.so.6 00002B7973FD6CAA abort Unknown Unknown
libstdc++.so.6 00002B7951ECBEA3 Unknown Unknown Unknown
libstdc++.so.6 00002B7951ED1E98 Unknown Unknown Unknown
libstdc++.so.6 00002B7951ED1EE3 Unknown Unknown Unknown
libstdc++.so.6 00002B7951ED2113 Unknown Unknown Unknown
libstdc++.so.6 00002B7951ED25DC Unknown Unknown Unknown
libvtkRenderingOp 00002B7966CE92AB _ZN16vtkOpenGLTex Unknown Unknown
libvtkRenderingOp 00002B7966CE8AB3 _ZN16vtkOpenGLTex Unknown Unknown
libvtkRenderingOp 00002B7966C8D881 _ZN23vtkOpenGLPol Unknown Unknown
libvtkRenderingOp 00002B7966C1D910 _ZN25vtkComposite Unknown Unknown
libvtkRenderingOp 00002B7966C1A52D _ZN27vtkComposite Unknown Unknown
libvtkRenderingOp 00002B7966C485B6 _ZN14vtkOpenGLAct Unknown Unknown
libvtkPVVTKExtens 00002B7970A4BFE4 _ZN13vtkPVLODActo Unknown Unknown
libvtkPVVTKExtens 00002B7970A4C261 _ZN13vtkPVLODActo Unknown Unknown
libvtkRenderingCo 00002B796927CC00 _ZN11vtkRenderer2 Unknown Unknown
libvtkRenderingOp 00002B7966CD0288 _ZN17vtkOpenGLRen Unknown Unknown
libvtkRenderingOp 00002B7966CD5FAC _ZN17vtkOpenGLRen Unknown Unknown
libvtkPVVTKExtens 00002B7970AE3E16 _ZN20vtkIceTCompo Unknown Unknown
libvtkPVVTKExtens 00002B7970AE5542 Unknown Unknown Unknown
libIceTCore.so 00002B7971DA77B5 Unknown Unknown Unknown
libIceTCore.so 00002B7971DA740C Unknown Unknown Unknown
libIceTCore.so 00002B7971DA61BD icetGetCompressed Unknown Unknown
libIceTCore.so 00002B7971DAE889 icetSequentialCom Unknown Unknown
libIceTCore.so 00002B7971DAE1ED icetInvokeStrateg Unknown Unknown
libIceTCore.so 00002B7971D9FD18 icetDrawFrame Unknown Unknown
libvtkPVVTKExtens 00002B7970AE18E0 _ZN20vtkIceTCompo Unknown Unknown
libvtkRenderingOp 00002B7966C179E0 _ZN13vtkCameraPas Unknown Unknown
libvtkRenderingOp 00002B7966CD0145 _ZN17vtkOpenGLRen Unknown Unknown
libvtkRenderingCo 00002B796927937D _ZN11vtkRenderer6 Unknown Unknown
libvtkRenderingCo 00002B796927767F _ZN21vtkRendererC Unknown Unknown
libvtkRenderingCo 00002B7969284FD2 _ZN15vtkRenderWin Unknown Unknown
libvtkRenderingCo 00002B79692827DA _ZN15vtkRenderWin Unknown Unknown
libvtkRenderingOp 00002B7966CCC5DA _ZN21vtkOpenGLRen Unknown Unknown
libvtkPVClientSer 00002B796D9BF3FD _ZN15vtkPVRenderV Unknown Unknown
libvtkPVClientSer 00002B796D9B8E91 _ZN15vtkPVRenderV Unknown Unknown
libvtkPVServerMan 00002B79579C67B2 _Z22vtkPVRenderVi Unknown Unknown
libvtkClientServe 00002B79723F30F8 _ZN26vtkClientSer Unknown Unknown
libvtkClientServe 00002B79723F1A7D _ZN26vtkClientSer Unknown Unknown
libvtkClientServe 00002B79723EF94C _ZN26vtkClientSer Unknown Unknown
libvtkClientServe 00002B79723F2EE9 _ZN26vtkClientSer Unknown Unknown
libvtkPVServerImp 00002B796E334048 _ZN16vtkPVSession Unknown Unknown
libvtkPVServerImp 00002B796E331BF5 _ZN16vtkPVSession Unknown Unknown
libvtkPVServerImp 00002B796E32F510 _ZN16vtkPVSession Unknown Unknown
libvtkPVServerMan 00002B796DD44283 _ZN14vtkSMViewPro Unknown Unknown
libvtkPVServerMan 00002B796DD03D2C _ZN20vtkSMRenderV Unknown Unknown
libvtkPVServerMan 00002B796DD4571E _ZN14vtkSMViewPro Unknown Unknown
libvtkPVServerMan 00002B796DD44FD4 _ZN14vtkSMViewPro Unknown Unknown
libvtkPVServerMan 00002B796DD44E05 _ZN14vtkSMViewPro Unknown Unknown
libvtkPVServerMan 00002B796DD0D460 _ZN24vtkSMSaveScr Unknown Unknown
libvtkPVServerMan 00002B796DD09B41 _ZN24vtkSMSaveScr Unknown Unknown
libvtkPVServerMan 00002B796DD087B5 _ZN24vtkSMSaveScr Unknown Unknown
libvtkPVServerMan 00002B799C4131DF Unknown Unknown Unknown
libpython2.7.so.1 00002B796886CBA6 Unknown Unknown Unknown
libpython2.7.so.1 00002B796886A1CA PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B796886F0AE Unknown Unknown Unknown
libpython2.7.so.1 00002B7968869C1B PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B796886F0AE Unknown Unknown Unknown
libpython2.7.so.1 00002B7968869C1B PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B79687C5CCC Unknown Unknown Unknown
libpython2.7.so.1 00002B796878B1EA PyObject_Call Unknown Unknown
libpython2.7.so.1 00002B796886CBB6 Unknown Unknown Unknown
libpython2.7.so.1 00002B796886A1CA PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B796886F0AE Unknown Unknown Unknown
libpython2.7.so.1 00002B7968869C1B PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B796886F0AE Unknown Unknown Unknown
libpython2.7.so.1 00002B7968869C1B PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886F17E Unknown Unknown Unknown
libpython2.7.so.1 00002B7968869C1B PyEval_EvalFrameE Unknown Unknown
libpython2.7.so.1 00002B796886DC56 PyEval_EvalCodeEx Unknown Unknown
libpython2.7.so.1 00002B796886D724 PyEval_EvalCode Unknown Unknown
libpython2.7.so.1 00002B79688A44B3 PyRun_StringFlags Unknown Unknown
libpython2.7.so.1 00002B79688A4539 PyRun_SimpleStrin Unknown Unknown
libvtkPythonInter 00002B79584842CA _ZN20vtkPythonInt Unknown Unknown
libvtkPVPythonCat 00002B79554D8723 _ZN25vtkCPPythonS Unknown Unknown
libvtkPVCatalyst- 00002B79552C8E83 _ZN14vtkCPProcess Unknown Unknown
regesm.x 00000000035E4478 Unknown Unknown Unknown
regesm.x 000000000054186A mod_esmf_cop_mp_c 1559 mod_esmf_cop.F90
libesmf.so 00002B794DD47E29 _ZN5ESMCI13Method Unknown Unknown
libesmf.so 00002B794DD47D4B _ZN5ESMCI11Method Unknown Unknown
libesmf.so 00002B794DD47162 c_esmc_methodtabl Unknown Unknown
libesmf.so 00002B794DF0F090 esmf_attachmethod Unknown Unknown
libesmf.so 00002B794E5652B5 nuopc_modelbase_m Unknown Unknown
libesmf.so 00002B794DA34785 ZN5ESMCI6FTable1 Unknown Unknown
libesmf.so 00002B794DA37D35 ESMCI_FTableCallE Unknown Unknown
libesmf.so 00002B794DE4B2A5 ZN5ESMCI2VM5ente Unknown Unknown
libesmf.so 00002B794DA35B38 c_esmc_ftablecall Unknown Unknown
libesmf.so 00002B794E042061 esmf_compmod_mp_e Unknown Unknown
libesmf.so 00002B794E1E111F esmf_gridcompmod
Unknown Unknown
libesmf.so 00002B794E535862 nuopc_driver_mp_r Unknown Unknown
libesmf.so 00002B794DA34785 ZN5ESMCI6FTable1 Unknown Unknown
libesmf.so 00002B794DA37D35 ESMCI_FTableCallE Unknown Unknown
libesmf.so 00002B794DE4B2A5 ZN5ESMCI2VM5ente Unknown Unknown
libesmf.so 00002B794DA35B38 c_esmc_ftablecall Unknown Unknown
libesmf.so 00002B794E042061 esmf_compmod_mp_e Unknown Unknown
libesmf.so 00002B794E1E111F esmf_gridcompmod
Unknown Unknown
regesm.x 0000000000419DFA MAIN
110 regesm.F90
regesm.x 00000000004193EE Unknown Unknown Unknown
libc.so.6 00002B7973FC16E5 __libc_start_main Unknown Unknown
regesm.x 00000000004192F9 Unknown Unknown Unknown

Do I need to set ICET_USE_OPENGL to OFF explicitly in this case? It seems that error is triggered from ICET. It was set as ON.

sounds like a issue with different libraries being loaded than built with or something like that. set LD_DEBUG=libs environment variable then see if from the generated log is something odd pops out.

The name of the Python library variable has likely changed from 5.6.1 to 5.7.0. If you look at your CMakeCache.txt file, that will show you the Python libraries against which you are linking. That should match /glade/u/home/turuncu/progs/python-2.7.16/lib/libpython2.7.so.1.0.

Like @utkarsh.ayachit said, LD_DEBUG=libs will tell you which Python library is loaded at runtime. If that matches what is in CMakeCache.txt, then I am puzzled. If it does not, then you’ve found the problem.

It seems it matches. So, following is the output of

cat CMakeCache.txt | grep PYTHON

command. The only thing that I noticed that is PYTHON_UTIL_LIBRARY points to /usr/lib64/libutil.so. Except that one everything seems fine.

PARAVIEW_ENABLE_ PYTHON :BOOL=ON

PARAVIEW_FREEZE_ PYTHON :BOOL=OFF

PYTHON _ENABLE_MODULE_mpi4py.MPI:BOOL=ON

PYTHON _ENABLE_MODULE_mpi4py.dl:BOOL=ON

PYTHON _EXECUTABLE:FILEPATH=/glade/u/home/turuncu/progs/python-2.7.16/bin/python2

PYTHON _EXTRA_LIBS:STRING=

PYTHON _INCLUDE_DIR:PATH=/glade/u/home/turuncu/progs/python-2.7.16/include/python2.7

PYTHON _LIBRARY:FILEPATH=/glade/u/home/turuncu/progs/python-2.7.16/lib/libpython2.7.so

PYTHON _MODULE_mpi4py.MPI_BUILD_SHARED:BOOL=ON

PYTHON _MODULE_mpi4py.dl_BUILD_SHARED:BOOL=ON

PYTHON _UTIL_LIBRARY:FILEPATH=/usr/lib64/libutil.so

VTK_ENABLE_VTK PYTHON :BOOL=ON

VTK_NO_ PYTHON _THREADS:BOOL=ON

VTK_ PYTHON _VERSION:STRING=2

VTK_USE_SYSTEM_CINEMA_ PYTHON :BOOL=OFF

//MODIFIED property for variable: PARAVIEW_ENABLE_ PYTHON

PARAVIEW_ENABLE_ PYTHON -MODIFIED:INTERNAL=ON

//ADVANCED property for variable: PARAVIEW_FREEZE_ PYTHON

PARAVIEW_FREEZE_ PYTHON -ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _ENABLE_MODULE_mpi4py.MPI

PYTHON _ENABLE_MODULE_mpi4py.MPI-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _ENABLE_MODULE_mpi4py.dl

PYTHON _ENABLE_MODULE_mpi4py.dl-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _EXECUTABLE

PYTHON _EXECUTABLE-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _EXTRA_LIBS

PYTHON _EXTRA_LIBS-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _INCLUDE_DIR

PYTHON _INCLUDE_DIR-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _LIBRARY

PYTHON _LIBRARY-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _MODULE_mpi4py.MPI_BUILD_SHARED

PYTHON _MODULE_mpi4py.MPI_BUILD_SHARED-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _MODULE_mpi4py.dl_BUILD_SHARED

PYTHON _MODULE_mpi4py.dl_BUILD_SHARED-ADVANCED:INTERNAL=1

//ADVANCED property for variable: PYTHON _UTIL_LIBRARY

PYTHON _UTIL_LIBRARY-ADVANCED:INTERNAL=1

VTK_BUILD_ PYTHON _MODULES_DIR:INTERNAL=/glade/u/home/turuncu/progs/paraview-5.6.1/install_mesa/lib/python2.7/site-packages

//ADVANCED property for variable: VTK_ENABLE_VTK PYTHON

VTK_ENABLE_VTK PYTHON -ADVANCED:INTERNAL=1

VTK_INSTALL_ PYTHON _MODULES_DIR:INTERNAL=lib/python2.7/site-packages

//ADVANCED property for variable: VTK_NO_ PYTHON _THREADS

VTK_NO_ PYTHON _THREADS-ADVANCED:INTERNAL=1

//ADVANCED property for variable: VTK_ PYTHON _FULL_THREADSAFE

VTK_ PYTHON _FULL_THREADSAFE-ADVANCED:INTERNAL=1

VTK_ PYTHON _FULL_THREADSAFE:INTERNAL=OFF

//ADVANCED property for variable: VTK_ PYTHON _VERSION

VTK_ PYTHON _VERSION-ADVANCED:INTERNAL=1

//STRINGS property for variable: VTK_ PYTHON _VERSION

VTK_ PYTHON _VERSION-STRINGS:INTERNAL=2;2.7;3;3.3;3.4;3.5;3.6;3.7;

//ADVANCED property for variable: VTK_USE_SYSTEM_CINEMA_ PYTHON

VTK_USE_SYSTEM_CINEMA_ PYTHON -ADVANCED:INTERNAL=1

VTK_WRAP_ PYTHON :INTERNAL=ON

XDMF_WRAP_ PYTHON :INTERNAL=OFF

After couple of weeks struggling with the installation, i solved the problem at least. After installing ParaView with my Python installation and using custom installed llvm and mesa libraries, i could able to produce output in co-processing mode. The mesa libraries shipped with OS was not able to render output without X (i am still don’t know why). Anyway, thanks for your great help and interest. Regards,