code is hanging in the initialization

In Situ Support
1

I am trying to use catalyst in my simulation code. I have node properly set up for the initialization

{
  "catalyst": 
  {
    "scripts": 
    {
      "script1": 
      {
        "filename": 
        {
          "dtype":"char8_str",
          "number_of_elements": 21,
          "offset": 0,
          "stride": 1,
          "element_bytes": 1,
          "endianness": "little",
          "value": "catalyst_pipeline.py"
        },
        "args": 
        [
          
          {
            "dtype":"char8_str",
            "number_of_elements": 20,
            "offset": 0,
            "stride": 1,
            "element_bytes": 1,
            "endianness": "little",
            "value": "--channel-name=grid"
          }
        ]
      }
    }
  },
  "catalyst_load": 
  {
    "implementation": 
    {
      "dtype":"char8_str",
      "number_of_elements": 9,
      "offset": 0,
      "stride": 1,
      "element_bytes": 1,
      "endianness": "little",
      "value": "paraview"
    },
    "search_paths": 
    {
      "paraview": 
      {
        "dtype":"char8_str",
        "number_of_elements": 78,
        "offset": 0,
        "stride": 1,
        "element_bytes": 1,
        "endianness": "little",
        "value": "/glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst"
      }
    }
  }
}

but calling initialization cause hanging in my code. I also enabled export CATALYST_DEBUG=1 to debug and see if there is an error but it seems it is fine since I could see following in stderr

catalyst debug: implementation name from `catalyst_load/implementation`: paraview

catalyst debug: search path from `catalyst_load/search_paths`: `/glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst`

catalyst debug: preferring environment variables? no

catalyst debug: trying to load `/glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst/libcatalyst-paraview.so`: valid

catalyst debug: loaded implementation: 0x149ff4c384b0

I wonder at this point if there is any way to debug it further or any suggestion.

2

I could able to collect backtrace by attaching gdb to the hanging processes

#0 0x0000152728608527 in sched_yield () from /lib64/libc.so.6
#1 0x000015272c47dade in MPIR_Wait_impl.part.0 () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#2 0x000015272d219a16 in MPIC_Wait () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#3 0x000015272d21fb02 in MPIC_Recv () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#4 0x000015272d13cd3b in MPIR_Bcast_intra_binomial () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#5 0x000015272b6c6696 in MPIDI_POSIX_mpi_bcast.isra.0.constprop.0 () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#6 0x000015272b77f5fc in MPIR_Allreduce () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#7 0x000015272d2590fc in MPIR_Get_contextid_sparse_group () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#8 0x000015272d25700c in MPII_Comm_copy () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#9 0x000015272b9e3c01 in MPIR_Comm_dup_impl () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#10 0x000015272b9e3e44 in PMPI_Comm_dup () from /opt/cray/pe/mpich/8.1.25/ofi/intel/19.0/lib/libmpi_intel.so.12
#11 0x00001526cd781dcc in vtkMPICommunicator::Duplicate(vtkMPICommunicator*) () from /glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst/../libvtkParallelMPI-pv5.13.so.1
#12 0x00001526cf313d94 in vtkInSituInitializationHelper::Initialize(unsigned long long) () from /glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst/../libvtkPVInSitu-pv5.13.so.1
#13 0x00001526cf52ee21 in catalyst_initialize_paraview () from /glade/work/turuncu/COP/progs/paraview-superbuild-5.13.2/install/lib/catalyst/libcatalyst-paraview.so
#14 0x00001527292b7939 in catalyst_initialize () from /glade/work/turuncu/COP/progs/catalyst-2.0.0/install/lib64/libcatalyst.so.3
#15 0x000000000401eb67 in cop_phases_catalyst::cop_phases_catalyst_run (gcomp=<error reading variable: Cannot access memory at address 0x0>, rc=<error reading variable: Cannot access memory at address 0x0>) at /glade/work/turuncu/COP/ufs-weather-model/COP/plugins/catalyst/cop_phases_catalyst.F90:159
3

Maybe it is hanging since I have no MPI communicator defined in the node (“catalyst/mpi_comm”). So, let me try that first.

4

Okay. It helped to pass the initialization. Thanks.