Loading tilted cartesian datasets

I am rather new to paraview, and I have a rather complicated challenge. I will divide this topic in three parts: (1) some background, (2) summary of data sets and (3) what I am trying to do
Some background:
We have a code, suitable for exa-scale supercomputing, which provides data in semi-disconnected patches (each patch is locally Cartesian), the simulation setup can have different, non-cartesian geometries. For example, now I am running simulations, where I have Cartesian patches arranged in almost a spherical arrangement, they have small overlaps. Each of these patches have their own local cartesian meshes (let’s say 20x20x20 cells) and a rotation matrix to know, where these cells would end up in a global reference frame.
My current lightweight development setup has about 25000 of such patches, arranged in 5 spherical shells. Production runs could have orders of magnitude more patches.

In short, the data set includes:
a large series of patches in some arbitrary arrangement;
each of these patches have central positions in global reference frame
each of these patches have local Cartesian system, with cell positions being in reference to the patch centre
each patch has a rotation matrix to know the orientation of the patch in a global domain

combining the local positions with global positions and the rotation matrix, I can have exact positions of each cell in the global reference frame ( matmul(e_rot, loc_pos)+glob_pos )

Loading all the data as a single point would be a terrible idea, of course.

My question is - is there an efficient way to load such data sets, e.g. using vtkPartitionedDataSet? Ideally I would like to load all the Cartesian patches and give them a position and tilt with respect to the centre of the global frame of reference.

Lastly, since these patches are fully locally Cartesian, they overlap each other in the global domain. Would Paraview handle the overlaps correctly? E.g. using data just from one source? Or should I expect glitches at such interfaces?

Thank you very much in advance,

There is nothing in VTK/ParaView (yet) . This was prototyped but not integrated afaik @Will_Schroeder .

You can of course do something custom in a plugin.

I’m not sure if this is getting to helping solve the problem, but it would be possible to at least represent your data by storing each partition as a vtkStructuredData. This type of data set still has the i, j, k reference of a Cartesian grid, but the points of the grid can be placed in general position with an array of point coordinates. Thus, you could apply your rotations to the initial Cartesian coordinates to get their rotated positions.

That said, this representation isn’t perfect. Storing the location of each point will take significantly more memory than just the rotations. Also, there will be some operations that VTK/ParaView could do on a Cartesian grid that it won’t do on the vtkStructuredData because it can no longer determine axis alignment or orthogonal hexahedrons.

Thanks for the reply! Unfortunately speed is utter important in my case, as in the production runs there will be terabytes of data, a snapshot consisting of hundreds of gigabytes. From my experience custom plugins are usually rather slugish, as they utilize sub-ideal solutions. Perhaps it would be possible to somewhere find the prototype, mayhaps I could utilize something from there

I was considering just that, but indeed, the sheer amount of extra memory gives me dread.

Completely untrue for C++ plugin, debatable for python plugin, which I’d not recommend in your case anyway.

ok, you got my interest. any recommendations where I could look for potential examples?

They are many examples in paraview source (Examples/Plugin).