I’ve experienced such a problem: so far I have the octree-structured data represented with separate cells in hdf5/xdmf format. The thing is, handling large data in such a format causes significant RAM blowups. Is there native octree support (or some thrid-party plugins) which would allow reducing the amount of memory needed to work with all that stuff (for instance, represent it in some other way, not storing all the cells as separate pieces of data)? It would be great to preserve the current format, but if it’s impossible, any other solution is welcome.
Are you looking for an AMR based dataset ?
VTK’s HyperTreeGrid data type could be a closer match to your format (tree based rather than the block structured AMR that Mathieu suggested).
Note that the HTG code prior to 5.6 didn’t get sufficient attention from users or developers and grew very buggy over time. We are actively working on it once more though and have fixed a number of bugs and added some new features, so 5.7 could be a good time to try it. See the VTK level regression tests and the Random Source in ParaView for some idea of what it is like.