In-Transit Catalyst implementation based on ADIOS2

To extend Catalyst (1) with In-Transit capabilities, we are designing an implementation based on the ADIOS2 library (2).


Catalyst is the favored tool to perform in-situ analysis with ParaView, running your pipeline on the fly without read/write on disk.

As of today, it does not allow In-Transit operations, i.e. moving data to dedicated MPI ranks for analysis.

ADIOS2 is “A Unified High-performance I/O Framework”. It supports different ways of communication, known as engine. The Sustainable Staging Transport (SST, 3) is the “Streaming-oriented” engine where “the number of reader ranks can differ from the number of writer ranks”, which is typically what we want here.


This work is splitted in two parts.

On the first side, we should create an ADIOS2 implementation of the Catalyst API. The simulation will use it as any Catalyst implementation. This will run on the Simulation nodes and any code already Catalyst-capable will benefit from this catalyst-adios variant without any need of further development.

On the Analysis nodes, we will have some kind of “catalyst replay” executable, to read the ADIOS2 streamed data and run the ParaView analysis.

As ADIOS2 and Conduit are both data descriptor, our goal is to convert from Conduit to ADIOS2 in the library and from ADIOS2 to Conduit in the “replay” executable, delegating the actual VTK objects creation to the existent code (vtkConduitSource).


@utkarsh.ayachit @chuckatkins Is it a reasonable approach ?

@Lucas_Givord @Francois_Mazen , for your information.

That’s seems consistent with what we had originally discussed. One thing to keep in mind, keep the condiut-2-adios as non-vtk as possible. There’s no reason why it should do any special processing of the data being passed and instead simply rely on conduit’s serialization infrastucture which we already use for catalyst replay.

Conduit itself already supports ADIOS serialization, IIRC. Our fork in Catalyst may have removed those parts, but it may be worth while investigating if we should just simply bring that back in.

1 Like