HDF5 has had the Single Writer Multiple Reader (SWMR) capability since version 1.10.
This feature allows to read HDF5 files properly while they’re being written and the file is not yet closed, but the relevant data structures are flushed.
The problem arises because like the command to write needs to specify that it’s an SWMR write via a flag, the readers need to provide this flag as well.
Paraview should consider this possibility so HDF5 files can be read while they’re being written, assuming that the XDMF file points only to already flushed data structures.
The topic is tagged with vtkhdf
, but the post mention xdmf
. Both use the HDF5 file format as the underlying data container, but as far as I am aware, these are completely separate implementations in VTK/Paraview.
Yes as @hakostra said that, xdmf and vtkhdf are incompletely different code, AFAIK both don’t support SWMR.
For VTKHDF if you’re interested you can write an issue in vtk to propose the support if if if you want https://gitlab.kitware.com/vtk/vtk/-/issues
FYI @danlipsa
Thank you!
I can provide a C++ code that generates 4 HDF5 files in parallel written with SWMR that keep adding timesteps without closing the file, and a Python code that opens them with SWMR, determines what timesteps have subdomains that are complete in that everything is flushed and writes the corresponding XDMF file for those timesteps and subdomains.
Of course, Paraview 5.10.0-RC1 crashes when trying to open the file without SWMR support, since it deems the “file is already open for write”. On the other hand, once the HDF5 is closed, the XDMF read works as expected.
I just tested adding the flag H5F_ACC_SWMR_READ
to each instance of H5Fread in the files
VTK/ThirdParty/xdmf3/vtkxdmf3/core/XdmfHDF5Controller.cpp
VTK/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfHDF.cxx
and it works!
I do not know if this affects the non-SWMR reading of HDF5 files through XDMF though.
Maybe some logic should be added to detect the presence of a SWMR property in the DataItem object of the XDMF file? Like this:
<DataItem Format="HDF" DataType="Float" Dimensions="100" SWMR="true">