Debugging segfault while passing data to Catalyst 2.1

In Situ Support
1

I’m passing a small 3D vector field (less than 100^3) to the Catalyst API. When the code runs single-threaded, and there is only one full grid to pass, things work without issue and the data correctly ends up in Paraview. But when switching to multi-threaded code, and e.g. 3 subgrids get passed, I get a segfault in the simulation code, that I’m trying to figure out.

(gdb) bt
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660
#1  0x00007ffff752e33d in catalyst_conduit::utils::default_memcpy_handler (destination=0x55555a9ab9f0, source=0x55555ab08760, num=1428840) at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_utils.cpp:106
#2  0x00007ffff75498df in std::__invoke_impl<void, void (*&)(void*, void const*, unsigned long), void*, void const*, unsigned long> (
    __f=@0x555555cc2328: 0x7ffff752e312 <catalyst_conduit::utils::default_memcpy_handler(void*, void const*, unsigned long)>) at /usr/include/c++/16/bits/invoke.h:63
#3  0x00007ffff754347f in std::__invoke_r<void, void (*&)(void*, void const*, unsigned long), void*, void const*, unsigned long> (
    __fn=@0x555555cc2328: 0x7ffff752e312 <catalyst_conduit::utils::default_memcpy_handler(void*, void const*, unsigned long)>) at /usr/include/c++/16/bits/invoke.h:152
#4  0x00007ffff753d51e in std::_Function_handler<void(void*, void const*, unsigned long), void (*)(void*, void const*, unsigned long)>::_M_invoke (__functor=..., __args#0=@0x7fffffffb220: 0x55555a9ab9f0, 
    __args#1=@0x7fffffffb218: 0x55555ab08760, __args#2=@0x7fffffffb210: 1428840) at /usr/include/c++/16/bits/std_function.h:295
#5  0x00007ffff753a019 in std::function<void(void*, void const*, unsigned long)>::operator() (this=0x555555cc2328, __args#0=0x55555a9ab9f0, __args#1=0x55555ab08760, __args#2=1428840) at /usr/include/c++/16/bits/std_function.h:581
#6  0x00007ffff7537d18 in catalyst_conduit::utils::detail::AllocManager::memcpy (this=0x555555cc22c0, destination=0x55555a9ab9f0, source=0x55555ab08760, num=1428840)
    at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_utils.cpp:201
#7  0x00007ffff752e69c in catalyst_conduit::utils::conduit_memcpy_strided_elements (dest=0x55555a9ab9f0, num_elements=357210, ele_bytes=4, dest_stride=4, src=0x55555ab08760, src_stride=4)
    at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_utils.cpp:315
#8  0x00007ffff74c9751 in catalyst_conduit::Node::set_float32_array (this=0x55555a0a3b60, data=...) at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_node.cpp:1124
#9  0x00007ffff74c9827 in catalyst_conduit::Node::set (this=0x55555a0a3b60, data=...) at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_node.cpp:1136
#10 0x00007ffff74ccc31 in catalyst_conduit::Node::set_float32_ptr (this=0x55555a0a3b60, data=0x55555ab08760, num_elements=187792, offset=0, stride=4, element_bytes=4, endianness=0)
    at /home/melis/c/catalyst/thirdparty/conduit/conduit/conduit_node.cpp:2492
#11 0x00007ffff756eb91 in catalyst_conduit_node_set_float32_ptr (cnode=0x55555a0a3b60, data=0x55555ab08760, num_elements=187792) at /home/melis/c/catalyst/thirdparty/conduit/conduit/c/conduit_node_c.cpp:956
#12 0x00005555555b11ca in conduit_cpp::Node::set_float32_vector (this=0x7fffffffb6f0, data=std::vector of length 187792, capacity 187792 = {...}) at /home/melis/software/catalyst/include/catalyst-2.1/catalyst_conduit.hpp:481
#13 0x00005555555b11f3 in conduit_cpp::Node::set (this=0x7fffffffb6f0, data=std::vector of length 187792, capacity 187792 = {...}) at /home/melis/software/catalyst/include/catalyst-2.1/catalyst_conduit.hpp:481

The relevant piece of code that sets the conduit values is shown below. The line triggering the segfault is fields[name+"/values/z"].set(zvalues); The grid sizes are not nice powers-of-2 (or even even, e.g. 44x97x44 or 187792 elements in the case above), but I assume that would not cause an issue like this. Also, since the code trips in memmove-vec-unaligned-erms.S I assume no specific memory-alignment is needed for the buffers being passed.

I have double-checked the amount of memory reserved, plus that it matches the dimensions passed through coordsets/coords/dims, although I assume the .set() is fully independent and doesn’t check the vector size passed versus the expected dimensions. So I think that’s not what’s going wrong, although there most likely is still an error on my side.

Looking at the strack trace it seems the num_elements value in #7 is double what it should be, and this causes an out-of-bounds access in #0.

Is there something I should do different in passing a std::vector<float> to Conduit?

This is with linking the sim code against Catalyst 2.1, running with Paraview 6.1.0 RC1.

  auto state = exec_params["catalyst/state"];
  state["timestep"].set(iT);
  state["time"].set(physTime);

  // We only have 1 channel here. Let's name it 'grid'.
  auto channel = exec_params["catalyst/channels/grid"];
  channel["type"].set("mesh");
  auto mesh = channel["data"];

  // Coordinate set (named "coords", referenced in the topology)
  mesh["coordsets/coords/type"].set("uniform");
  mesh["coordsets/coords/dims/i"].set(extent1[0]+2*_overlap);
  mesh["coordsets/coords/dims/j"].set(extent1[1]+2*_overlap);
  mesh["coordsets/coords/dims/k"].set(extent1[2]+2*_overlap);
  mesh["coordsets/coords/origin/x"].set(originPhysR[0]);
  mesh["coordsets/coords/origin/y"].set(originPhysR[1]);
  mesh["coordsets/coords/origin/z"].set(originPhysR[2]);
  mesh["coordsets/coords/spacing/dx"].set(delta);
  mesh["coordsets/coords/spacing/dy"].set(delta);
  mesh["coordsets/coords/spacing/dz"].set(delta);

  mesh["topologies/topo/type"].set("uniform");
  mesh["topologies/topo/coordset"].set("coords");

  auto fields = mesh["fields"];
  fields[name+"/association"].set("vertex");
  fields[name+"/topology"].set("topo");
  fields[name+"/volume_dependent"].set("false");

  size_t num_elements = (extent1[0] + 2*_overlap) * (extent1[1] + 2*_overlap) * (extent1[2] + 2*_overlap);
  size_t num_values = num_elements * components;

  std::vector<float> xvalues, yvalues, zvalues;

  xvalues.reserve(num_elements);
  yvalues.reserve(num_elements);
  zvalues.reserve(num_elements);

  for (i[3] = -_overlap; i[3] < extent1[2]+_overlap; ++i[3]) {
    for (i[2] = -_overlap; i[2] < extent1[1]+_overlap; ++i[2]) {
      for (i[1] = -_overlap; i[1] < extent1[0]+_overlap; ++i[1]) {
        (*functor)(evaluated, i);
        xvalues.push_back(float(evaluated[0]));
        yvalues.push_back(float(evaluated[1]));
        zvalues.push_back(float(evaluated[2]));
      }
    }
  }

  fields[name+"/values/x"].set(xvalues);
  fields[name+"/values/y"].set(yvalues);
  fields[name+"/values/z"].set(zvalues);
2

This looks like a data race - can you provide more details around how the threads are dispatched? Without knowing the specifics I would keep the conduit node construction serial and just do the subgrid build parallel - building the node is cheap, and conduit::Node is not threadsafe within one root because child node writes can modify parent nodes in the tree.

struct SubgridData {
    std::string name;
    int dims[3];
    double origin[3], delta;
    std::vector<float> x, y, z;   // filled in parallel
};

// PARALLEL: threads only touch their own SubgridData, no Conduit calls
std::vector<SubgridData> grids(num_subgrids);
#pragma omp parallel for
for (int g = 0; g < num_subgrids; ++g)
    fill_subgrid(grids[g]);       // your loop + functor -> x/y/z

// SERIAL: one thread builds the tree and executes
for (const auto& sg : grids) {
    auto channel = exec_params["catalyst/channels/" + sg.name];
    channel["type"].set("mesh");
    auto mesh = channel["data"];
    // ... coordsets / topo as before, from sg.dims/origin/delta ...
    auto fields = mesh["fields"];
    fields[sg.name + "/values/x"].set(sg.x);
    fields[sg.name + "/values/y"].set(sg.y);
    fields[sg.name + "/values/z"].set(sg.z);
}
catalyst_execute(conduit_cpp::c_node(&exec_params));
3

Hmm, thanks, that’s very helpful and something I’m going to look into.