We got the following error saving a slice of a large case with Catalyst. The error seems to be coming from the CGNS writer, which would have been called by the CGNS Extractor.
vtkMPICommunicator.cxx:205 WARN| This operation not yet supported for more than 2147483647 objects
Note that 2147483647 is int max.
I see that we have a #define VTKMPI_64BIT_LENGTH. Would that allow us to pass longer messages? I’m surprised that it is a #define and not a cmake option. Is the use of this feature recommended?
Note that #define VTKMPI_64BIT_LENGTH is defined only when MPI_VERSION >= 4 in vtkMPICommunicator.cxx.
The message size limit is a limitation with MPI implementations that conform to an MPI version less than 4.0. Implementations of MPI 4.0 or above support large messages. Are you building against such an MPI?
If not, updating to MPI 4.0 or higher should fix this issue automatically, as the large message versions of calls should be used instead.
If you can’t update to MPI 4.0, another approach is to break the failing communication call into several calls, each of which transfers less than the maximum number of items that can be communicated in a single call. This would need to be done in the reader and can get complicated depending on the communication pattern, but it is possible at least.
Yep, I thought about that, though if we were to modify the CGNS writer, I think we would switch to parallel CGNS write calls instead of serializing the output. I really don’t expect to run into that issue often. The idea of in situ is to create small datasets from large ones, so there are plenty of work arounds.