The VTU file was generated from a in-house CFD solver based on the High Lift Workshop benchmark case. It contains 26M grid points and ~65M cells. The files are placed at Google drive: volumes_hlcrm_medium.tar - Google Drive
This VTU file can not be loaded into Paraview (Windows version) due to the error message:
ERROR: In vtkXMLDataParser.cxx, line 651
vtkXMLDataParser (000002913777F810): Error reading uncompressed binary data header. Read 0 of 8 bytes.
ERROR: In vtkXMLUnstructuredDataReader.cxx, line 738
vtkXMLUnstructuredGridReader (0000029150A5EC70): Cannot read cell offsets from Cells in piece 0 because the “offsets” array is not long enough.
I tried 5.11.1, 5.10.1 and 5.9.1 versions of Paraview, the error messages are the same. The strange thing is the above file could be loaded into Linux’s version of Paraview successfully.
Could anyone take a look at this issue? Thank you!
This problem seems to keep resurfacing in different guises on Windows.
I have a program in c++ that generates the xml file based on other input data. It has been working for years for many different inputs. This is the first time the xml file exceeds 2GB. https://gitlab.kitware.com/paraview/paraview/-/issues/19628
Hi @todoooo , thanks for these useful links. In recent days, I found the same thing as your link: https://gitlab.kitware.com/paraview/paraview/-/issues/21145.
I also track down at the XMLByteIndex, which is a abnormal negative number at Windows (~1.7 billion), but it is a normal number at Linux (2.5 Billion, which is bigger than max(Int32_t), but I confirmed that vtkTypeId is 64-bit). Then I found the VTK library uses expat to parse the XML documentation. You mentioned
Hard coding add_definitions(-DXML_LARGE_SIZE) in the CMakeLists.txt file for the IOXMLParser project resolves this
Could you tell me more about this? Also, do you know how to rebuild the entire Paraview release from source code at Windows OS? I found a repo called Paraview-superbuild, but not sure about whether it is the correct way: https://gitlab.kitware.com/paraview/paraview-superbuild
Basically what is happening is that the XML 3rd party library is being built with a stream size of type Long. On 64 bit Linux/Mac that maps to 8 bytes, but on Win64 it maps to 4 bytes, so once the stream position exceeds ~2Gb the index becomes negative. The way to fix this is to force the compiler to define XML_LARGE_SIZE when building the XML library for 64 bit. This should be configured in CMake but I couldn’t figure out how to implement it.
If you find the IOXMLParser folder in the Paraview source and add the XML_LARGE_SIZE definition to its CMakeLists file you have a hacky work-around solution.
You can build Paraview from the Gitlab master branch (or a release branch) on Windows by configuring CMake for Visual Studio or Ninja. I use Visual Studio. I have never tried the super-build but it is probably similar.
@todoooo Thanks! Yup, the “long” at Windows is 32-bit. I’ve found the XML_LARGE_SIZE in the VTK/ThirdParty/expat/vtkexpat/CMakeLists.txt library. Currently it is OFF. I will make it ON to have a quick test.
@todoooo Do you know what the compilation options used by the official released Paraview at https://www.paraview.org/download/? I want to build a version using the same options as the official version with only the change of the above XML_LARGE_SIZE. Thanks.
@todoooo I’ve tried to turn on the EXPAT_LARGE_SIZE at the expat’s library, but the built VTK library still fails to read my big VTU file with throwing out the same error message.
In the VTK/ThirdParty/expat/vtkexpat/CMakeLists.txt, I made the change to turn on the EXPAT_LARGE_SIZE:
option(EXPAT_LARGE_SIZE “Make XML_GetCurrent* functions return <(unsigned) long
long> rather than <(unsigned) long>” ON)
I also confirmed that the “-DXML_LARGE_SIZE” was added into definitions, since I added a log into the above CMakeLists.txt file. Since this issue occurs in the level of VTK library, I didn’t compile the entire Paraview application. I just compiled the VTK library and let it to read my VTU file using one of its unit tests (its path is VTK\IO\XML\Testing\Cxx\TestXMLUnstructuredGridReader.cxx), but it still gives me the same error like before.
I guess I have to dive into the source code of the XML parser to figure it out…
@todoooo I confirmed that your solution works by adding add_definitions(-DXML_LARGE_SIZE) into ParaviewSrc\VTK\IO\XMLParser\CMakeLists.txt. Previously, I only added the above definition into the vtkexpat part, which is not enough. With your solution, now I can read the big VTU file into Paraview smoothly. Thanks.
Please look for an existing ParaView issue that describes this problem, or file a new one, and describe your solution. This should be pretty easy to get fixed. If you can provide a data file that causes the error on Windows, we can confirm it is fixed, too.
@cfd_ow The change I suggested should be sufficient. AFAIK editing any of the VTK third party library cmake files has no effect on their building in Paraview.
@cfd_ow : Thanks for opening the issues and identifying a working fix. Lets hope someone will actually tackle this and fix it. Do not hesitate to do this yourself if you feel like it.
I would do that if I knew how it was supposed to work, but it was the lack of comprehension that prevented me from creating an MR fix in the first place.
