can't read ADIOS2 BP5 file

ParaView Support
1

I have ADIOS2 files written using the ADIOS2 BP5 engine.

However, Paraview 5.12.0-RC1 can’t read them. It gives this error:

ERROR: In vtkADIOS2CoreImageReader.cxx, line 397
vtkADIOS2CoreImageReader (0000024CCA07C450): failed to open and read meta data: e[1;36m[Sun Dec 17 12:51:35 2023]e[1;34m [ADIOS2 EXCEPTION]e[0m <Core> <IO> <Operator> : ERROR: this version didn't compile with BP5 library, can't use BP5 engine
e[0m


ERROR: In vtkADIOS2CoreImageReader.cxx, line 431
vtkADIOS2CoreImageReader (0000024CCA07C450): unable to open file and data

ERROR: In vtkExecutive.cxx, line 730
vtkPVCompositeDataPipeline (0000024CCA2D4840): Algorithm vtkADIOS2CoreImageReader (0000024CCA07C450) returned failure for request: vtkInformation (0000024CD2D53E10)
  Debug: Off
  Modified Time: 444176
  Reference Count: 1
  Registered Events: (none)
  Request: REQUEST_INFORMATION
  FORWARD_DIRECTION: 0
  ALGORITHM_AFTER_FORWARD: 1

I assume this is a bug. Is there a version of Paraview that was compiled with BP5 support?

2

Hello @BenWibking,

I assume last RC should be able to open an BP5 file since this merge request : https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10152.

It may be related to this issue about BP4, BP5 support : https://gitlab.kitware.com/paraview/paraview/-/issues/22108

@caitlin.ross wdyt?

3

@BenWibking , please share your file

4

Hmm that’s weird. So I just tested out the 5.12-RC1 binary for Mac and was able to open a BP5 file with the ADIOS2CoreImageReader without problem, so the ADIOS used to build the binaries do have BP5 support turned on.

What platform are you on? What version of ADIOS2 did you use to write the files?

I’ve definitely seen misleading error messages from ADIOS at times (like in the issue Lucas linked), so it seems that something else is probably happening but it’s not giving an error message at the real failure. If you can share the file, I can investigate it with a debugger.

5

I tested this on Windows. I used ADIOS2 2.9.2 to write the file. Here is the file:
plt00758.bp.zip (2.7 MB)

6

Thanks! I was able to open it on my Mac, so it is just a Windows issue. I looked in a build log and it seems that for some reason on Windows BP5 support isn’t being built, though I’m not sure why yet. As far as I can tell we’re not explicitly disabling it in our configuration. I know it took the ADIOS team a bit of time to get BP5 supported on Windows, but we’re using ADIOS2.9.2 when building ParaView, so we should be able to build it with that support. I’m guessing that our Windows machines must be missing a dependency needed to enable BP5 support so the CMake configuration is automatically silently turning it off since we don’t explicitly turn on that option. I’ll try to get it fixed in time to go into RC2.

7

Excellent, thanks!

8

I tried opening the same dataset on my Mac. Each individual file is opened properly, but when I open it as a timeseries, the time slider shows spurious times and timesteps, and does not allow me to examine anything except the first timestep/ADIOS2 iteration.

I get this error message:

ERROR: In vtkADIOS2CoreImageReader.cxx, line 505
vtkADIOS2CoreImageReader (0x291e1c490): The requested time step 1 is not available!

ERROR: In vtkExecutive.cxx, line 730
vtkPVCompositeDataPipeline (0x600001a0c600): Algorithm vtkADIOS2CoreImageReader (0x291e1c490) returned failure for request: vtkInformation (0x60000c2ad2c0)
  Debug: Off
  Modified Time: 762121
  Reference Count: 1
  Registered Events: (none)
  Request: REQUEST_DATA
  FROM_OUTPUT_PORT: 0
  ALGORITHM_AFTER_FORWARD: 1
  FORWARD_DIRECTION: 0

I’ve attached the full timeseries (only 2 timesteps/iterations) that I used to test this here:
adios2_bp5_files.tar.gz (2.7 MB)

9

So as far as I can tell by digging around in ADIOS CMake files, BP5 isn’t available for Windows in 2.9.2. Did you write the file on a non-Windows system? I originally assumed it was written on Windows, but I want to double check.

10

Ah, that’s unfortunate. I wrote the file on a Linux system.

11

So I think a 2.10 release is supposed to happen soon. The person who does the ADIOS2 releases is out today, but I’ll check with him tomorrow on when that is supposed to be. Anyway once that happens and we update ParaView to build with that version, you could download a nightly on Windows.

As for the time error, I did recently find some bugs in this reader. Those fixes will be in the next RC. I’ve got a release build going right now so I can test it with your file. If it still doesn’t work, I’ll try to get a fix in for it soon.

1 Like
12

Oh wait, I glossed over the part about each time step being in its own file. I don’t think we support that in any of the ADIOS readers (there’s 2 others besides the one you’re using). ADIOS has an open issue for a feature that would handle this (see this issue), but it’s quite old so I’m not sure if it’s still planned. We haven’t planned to support this directly in the ADIOS readers in ParaView, but I can double check on that.

Is there a reason you’re writing each step in its own file? Usually we recommend to write all time steps to a single file, since the file doesn’t get corrupted if your application crashes, but I’m sure there are other reasons for doing this.

13

This is mostly for backward compatibility, since our previous file format wrote to separate files for each timestep. It also makes it easier to decimate in time resolution for long-term storage or transfer over the internet.

Does the openPMD reader support separate files per timestep? We used openPMD’s ADIOS2 backend to write these files and selected the “file-based iteration encoding” (Concepts — openPMD-api 0.15.2 documentation). I’d hoped this would be supported by all of the tools that could read it.

14

I just tried opening the files with the openPMD reader on Mac, and it seems to work perfectly. I was thrown off by the fact that it didn’t work on Windows, but that was probably due to the fact that it couldn’t read the BP5 files themselves.

15

Okay good! Glad that works with the OpenPMD reader. As for the ADIOS readers, I’ll bring up the file series support with the ADIOS team, though it sounds like in your case you’ll just stick with the OpenPMD reader and not use the ADIOS Image reader.

I asked about the ADIOS 2.10 release and it’s going to start soon (in the next week). If that gets finished before the ParaView 5.12 release process is finished, we plan to switch that to use 2.10, in which case the OpenPMD reader and other ADIOS readers would be able to read BP5 files on Windows.

16

I am still unable to open BP5 files with the OpenPMD reader with Paraview 5.12-RC3 on Windows:

ERROR: In vtkPythonAlgorithm.cxx, line 97
vtkPythonAlgorithm (000001CA72FBD150): Failure when calling method: "ProcessRequest":

Traceback (most recent call last):
  File "C:\Program Files\ParaView 5.12.0\bin\Lib\site-packages\vtkmodules\util\vtkAlgorithm.py", line 152, in ProcessRequest
    return vtkself.ProcessRequest(request, inInfo, outInfo)
  File "C:\Program Files\ParaView 5.12.0\bin\Lib\site-packages\vtkmodules\util\vtkAlgorithm.py", line 194, in ProcessRequest
    return self.RequestInformation(request, inInfo, outInfo)
  File "C:\Program Files\ParaView 5.12.0\bin\Lib\site-packages\paraview\algorithms\openpmd.py", line 283, in RequestInformation
    self._series = io.Series(
ValueError: e[1;36m[Tue Feb 13 14:02:49 2024]e[1;34m [ADIOS2 EXCEPTION]e[0m <Core> <IO> <Operator> : ERROR: this version didn't compile with BP5 library, can't use BP5 engine
e[0m

ERROR: In vtkExecutive.cxx, line 730
vtkPVCompositeDataPipeline (000001CA69DBD670): Algorithm vtkPythonAlgorithm (000001CA72FBD150) returned failure for request: vtkInformation (000001CA734BF050)
  Debug: Off
  Modified Time: 422127
  Reference Count: 2
  Registered Events: (none)
  Request: REQUEST_INFORMATION
  FORWARD_DIRECTION: 0
  ALGORITHM_AFTER_FORWARD: 1
17

Unfortunately we won’t be able to support reading BP5 on Windows with any ADIOS readers (including OpenPMD) for 5.12.0. BP5 isn’t able to be used on Windows until ADIOS 2.10. There’s been a release candidate for 2.10, but the final release isn’t expected for another month or so, so we won’t be able to update ADIOS in ParaView until the next release.

18

Thanks for the update. I’ll mark it as solved.