Reading error with VTKHDF file with large number of timesteps

ParaView Support
1

Dear Support
I have converted a time-series of data with 10641 timesteps, for which I have already created an animation, so I know that I have valid data. I saved the data with the command
SaveData(‘/capstor/scratch/cscs/jfavre/xxx/xxx.vtkhdf’, proxy=reader, WriteAllTimeSteps=1)
there was no error shown while saving the data.
When reading the data back using the new format, all timesteps are good up until timestep 6740, and then, without any error messages, all timesteps show data filled with 0. I have tried reading it with v5.13 and 6.0-RC1
The data is rather large 114G, so not easy to share. The structure is:

/                        Group
/VTKHDF                  Group
/VTKHDF/Connectivity     Dataset {815616/Inf}
/VTKHDF/NumberOfCells    Dataset {10641/Inf}
/VTKHDF/NumberOfConnectivityIds Dataset {10641/Inf}
/VTKHDF/NumberOfPoints   Dataset {10641/Inf}
/VTKHDF/Offsets          Dataset {203905/Inf}
/VTKHDF/PointData        Group
/VTKHDF/PointData/acceleration Dataset {3390222600/Inf, 3}
/VTKHDF/PointData/displacement Dataset {3390222600/Inf, 3}
/VTKHDF/PointData/velocity Dataset {3390222600/Inf, 3}
/VTKHDF/Points           Dataset {318600/Inf, 3}
/VTKHDF/Steps            Group
/VTKHDF/Steps/CellOffsets Dataset {10641/Inf}
/VTKHDF/Steps/ConnectivityIdOffsets Dataset {10641/Inf}
/VTKHDF/Steps/PartOffsets Dataset {10641/Inf}
/VTKHDF/Steps/PointDataOffsets Group
/VTKHDF/Steps/PointDataOffsets/acceleration Dataset {10641/Inf}
/VTKHDF/Steps/PointDataOffsets/displacement Dataset {10641/Inf}
/VTKHDF/Steps/PointDataOffsets/velocity Dataset {10641/Inf}
/VTKHDF/Steps/PointOffsets Dataset {10641/Inf}
/VTKHDF/Steps/Values     Dataset {10641}
/VTKHDF/Types            Dataset {203904/Inf}

is there some known limitation, with number of timesteps, or size, or whatever?

2

Hello Jean, curiously 6740/10641 is nearly equal to 2147483647 (int32 limit)/3390222600… that sounds like we use an int32 for indexing somewhere we should be using an int64

3

Good math. Can it be fixed before the 6.0 release?

4

Could you please first make sure that:

  • VTK_USE_64BIT_IDS is turned on for your ParaView build
  • All of your offset arrays, including those in the Steps group, are in fact int64
5

grep VTK_USE_64BIT_IDS CMakeCache.txt
VTK_USE_64BIT_IDS:BOOL=ON
//ADVANCED property for variable: VTK_USE_64BIT_IDS
VTK_USE_64BIT_IDS-ADVANCED:INTERNAL=1

6

What is the output of h5dump -H on your file?

7

HDF5 “/capstor/scratch/cscs/jfavre/Salvus/salvus.vtkhdf” {
GROUP “/” {
GROUP “VTKHDF” {
ATTRIBUTE “Type” {
DATATYPE H5T_STRING {
STRSIZE 16;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE “Version” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
}
DATASET “Connectivity” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 815616 ) / ( H5S_UNLIMITED ) }
}
DATASET “NumberOfCells” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “NumberOfConnectivityIds” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “NumberOfPoints” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “Offsets” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 203905 ) / ( H5S_UNLIMITED ) }
}
GROUP “PointData” {
DATASET “acceleration” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3390222600, 3 ) / ( H5S_UNLIMITED, 3 ) }
}
DATASET “displacement” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3390222600, 3 ) / ( H5S_UNLIMITED, 3 ) }
}
DATASET “velocity” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3390222600, 3 ) / ( H5S_UNLIMITED, 3 ) }
}
}
DATASET “Points” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 318600, 3 ) / ( H5S_UNLIMITED, 3 ) }
}
GROUP “Steps” {
ATTRIBUTE “NSteps” {
DATATYPE H5T_STD_I64LE
DATASPACE SCALAR
}
DATASET “CellOffsets” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “ConnectivityIdOffsets” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “PartOffsets” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
GROUP “PointDataOffsets” {
DATASET “acceleration” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “displacement” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “velocity” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
}
DATASET “PointOffsets” {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 10641 ) / ( H5S_UNLIMITED ) }
}
DATASET “Values” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 10641 ) / ( 10641 ) }
}
}
DATASET “Types” {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 203904 ) / ( H5S_UNLIMITED ) }
}
}
}
}

8

I believe the culprit here is the writer. Could you please test this patch? I converted a bunch of int to long/vtkIdType https://gitlab.kitware.com/vtk/vtk/-/merge_requests/12195

9

I have tried the patch provided and it solves the issue. Thanks a bunch.

2 Likes
10

Neat! I’ll try to get it in ParaView 6.0

11

The first two changes you make, to getBlockName() and vtkHDFWriter::AppendBlocks(), changed int to unsigned int. While this change should double the number of block IDs, should these also be changed to vtkIdType?

12

The number of blocks in a composite data structure will never overflow an int32, you will get issues with the VTK executive way before. I converted to unsigned int to match the return type of vtkPartitionedDataSetCollection::GetNumberOfPartitionedDataSets() and avoid casts, but this is not the core fix here.