How to merge multiple PVD and VTP files

GWAVE · February 5, 2020, 8:56am

Does anyone know of a script to merge multiple PVD and VTP files? I’m talking 100s of…not 2 or 3.

Question details below…

Looking at one of the VTP files, by typing on the command line:

more PART_00000.vtp

I get the following output…

<?xml version='1.0'?>
<VTKFile type='PolyData'  version='0.1'  byte_order='LittleEndian'>
 <PolyData>
  <Piece NumberOfPoints='2221680' NumberOfVerts='2221680'>
   <Points>
	<DataArray type='Float64' Name='Position' NumberOfComponents='3' format='appended' offset='0'/>
   </Points>
   <PointData Scalars='Pressure' Vectors='Velocity'>
	<DataArray type='Float32' Name='Pressure' format='appended' offset='53320324'/>
	<DataArray type='Float32' Name='Velocity' NumberOfComponents='3' format='appended' offset='62207048'/>
	<DataArray type='Float32' Name='Density' format='appended' offset='88867212'/>
	<DataArray type='Float32' Name='Mass' format='appended' offset='97753936'/>
	<DataArray type='Float32' Name='SPS turbulent viscosity' format='appended' offset='106640660'/>
	<DataArray type='UInt8' Name='Part type' format='appended' offset='115527384'/>
	<DataArray type='UInt8' Name='Part flags' format='appended' offset='117749068'/>
	<DataArray type='UInt8' Name='Part object' format='appended' offset='119970752'/>
	<DataArray type='UInt32' Name='Part id' format='appended' offset='122192436'/>
	<DataArray type='UInt32' Name='CellIndex' format='appended' offset='131079160'/>
   </PointData>
   <Verts>
	<DataArray type='UInt32' Name='connectivity' format='appended' offset='139965884'/>
	<DataArray type='UInt32' Name='offsets' format='appended' offset='148852608'/>
   </Verts>
  </Piece>
 </PolyData>
 <AppendedData encoding='raw'>

followed by a bunch of binary/encoded stuff. I have a whole bunch of these types of files which can be found listed in a ParaView Data file (.pvd format) by executing the following from the command line

more VTUinp.pvd

which gives

<?xml version='1.0'?>
<VTKFile type='Collection' version='0.1'>
 <Collection>
  <DataSet timestep='0' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0' group='1' name='WaveGages' file='WaveGage_00000.vtu'/>
  <DataSet timestep='0' group='2' name='Particles' file='PART_00000.vtp'/>
  <DataSet timestep='0.1000990719476249' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.1000990719476249' group='1' name='WaveGages' file='WaveGage_00001.vtu'/>
  <DataSet timestep='0.1000990719476249' group='2' name='Particles' file='PART_00001.vtp'/>
  <DataSet timestep='0.2000853080899105' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.2000853080899105' group='1' name='WaveGages' file='WaveGage_00002.vtu'/>
  <DataSet timestep='0.2000853080899105' group='2' name='Particles' file='PART_00002.vtp'/>
  <DataSet timestep='0.3000886073423317' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.3000886073423317' group='1' name='WaveGages' file='WaveGage_00003.vtu'/>
  <DataSet timestep='0.3000886073423317' group='2' name='Particles' file='PART_00003.vtp'/>
  <DataSet timestep='0.4000112119101686' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.4000112119101686' group='1' name='WaveGages' file='WaveGage_00004.vtu'/>
  <DataSet timestep='0.4000112119101686' group='2' name='Particles' file='PART_00004.vtp'/>
  <DataSet timestep='0.5000988173051155' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.5000988173051155' group='1' name='WaveGages' file='WaveGage_00005.vtu'/>
  <DataSet timestep='0.5000988173051155' group='2' name='Particles' file='PART_00005.vtp'/>
  <DataSet timestep='0.6000160886978847' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.6000160886978847' group='1' name='WaveGages' file='WaveGage_00006.vtu'/>
  <DataSet timestep='0.6000160886978847' group='2' name='Particles' file='PART_00006.vtp'/>
  <DataSet timestep='0.700017443698016' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.700017443698016' group='1' name='WaveGages' file='WaveGage_00007.vtu'/>
  <DataSet timestep='0.700017443698016' group='2' name='Particles' file='PART_00007.vtp'/>
  <DataSet timestep='0.8000870649775607' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.8000870649775607' group='1' name='WaveGages' file='WaveGage_00008.vtu'/>
  <DataSet timestep='0.8000870649775607' group='2' name='Particles' file='PART_00008.vtp'/>
  <DataSet timestep='0.9000359851925168' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='0.9000359851925168' group='1' name='WaveGages' file='WaveGage_00009.vtu'/>
  <DataSet timestep='0.9000359851925168' group='2' name='Particles' file='PART_00009.vtp'/>
  <DataSet timestep='1.000061886152253' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.000061886152253' group='1' name='WaveGages' file='WaveGage_00010.vtu'/>
  <DataSet timestep='1.000061886152253' group='2' name='Particles' file='PART_00010.vtp'/>
  <DataSet timestep='1.100044853170402' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.100044853170402' group='1' name='WaveGages' file='WaveGage_00011.vtu'/>
  <DataSet timestep='1.100044853170402' group='2' name='Particles' file='PART_00011.vtp'/>
  <DataSet timestep='1.200035295594716' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.200035295594716' group='1' name='WaveGages' file='WaveGage_00012.vtu'/>
  <DataSet timestep='1.200035295594716' group='2' name='Particles' file='PART_00012.vtp'/>
  <DataSet timestep='1.300029617945256' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.300029617945256' group='1' name='WaveGages' file='WaveGage_00013.vtu'/>
  <DataSet timestep='1.300029617945256' group='2' name='Particles' file='PART_00013.vtp'/>
  <DataSet timestep='1.400022054469446' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.400022054469446' group='1' name='WaveGages' file='WaveGage_00014.vtu'/>
  <DataSet timestep='1.400022054469446' group='2' name='Particles' file='PART_00014.vtp'/>
  <DataSet timestep='1.500012752476323' group='0' name='Topography' file='DEM.vts'/>
  <DataSet timestep='1.500012752476323' group='1' name='WaveGages' file='WaveGage_00015.vtu'/>
  <DataSet timestep='1.500012752476323' group='2' name='Particles' file='PART_00015.vtp'/>
 </Collection>
</VTKFile>

In Paraview, when I go to open a single group of files it gives me the option to open Part_…vtp which opens all the files contained in that directory (those listed in the .pvd file above) and I can step through them without a problem. Great!

My problem is that the program that creates these files sometimes needs to perform a “resume/restart/hotstart”. When this happens, a new output directory is created, so I have several directories containing 1) Many Part*.vtp files 2) Corresponding VTUinp.pvd 3) Plus bunch of other files. The restart of the filename indexing isn’t really a problem, because the ParaView Data file thankfully tracks the time step. However, I want to open the full time series (all “resume/restart/hotstart” files) in ParaView, but I don’t know how to do this. Ideally I would like to merge the necessary data files with a script before even trying to open in ParaView. I think there is a “GroupData” filter, but I have 100s of folders and it will take forever to do that by hand (not to mention doing it more than once).

Can someone tell me if such a script exists? If not, can someone give me some insight on how I would do this?

mwestphal · February 5, 2020, 9:04am

Open all your files, then :

GroupDatasets(Input=list(GetSources().values()))

GWAVE · February 5, 2020, 9:35am

This is precisely what I said I wanted to avoid in the last sentence of the second to last paragraph “I think there is a “GroupData” filter, but I have 100s of folders and it will take forever to do that by hand (not to mention doing it more than once).”

mwestphal · February 5, 2020, 10:33am

Open them using a for loop in python, not in the GUI.

GWAVE · February 5, 2020, 10:32pm

Ahhh…this sounds promising. Thank you for the insight! I’m new to ParaView so I’m not fully aware of how to use it beyond the GUI. I’ve read in passing that there are many things that can be done through scripting but I have yet to venture down this path. I guess it is time.

I’m sure I can research this functionality on my own, but do you have a good example/tutorial you can share/reference to help me get started?

mwestphal · February 6, 2020, 2:18am

 import os                                                                   
 import glob                                                                 
 os.chdir('path/to/your/folder')         
 for file in list(glob.glob('*.pvd')):                                       
     OpenDataFile(file)
 for file in list(glob.glob('*.vtp')):                                       
     OpenDataFile(file)

https://kitware.github.io/paraview-docs/latest/python/
https://www.paraview.org/Wiki/ParaView#Python_Scripting