I need to use PV to visualize/analyze >1TB data sets (900M cell structured grid time series) in which each stored data frame is approximately 50GB.
I am preparing by making sure I can efficiently access a much smaller (1.6GB/frame) test dataset. However, I have now tried reading the same data stored in Ensight Gold, .vtk, and .h5+xdmf and all take hugely longer than my media speed would imply, whether in serial or connecting to 1/2/4 pvservers on my laptop.
I have a 4 (physical) core Xeon, 32GB of DDR4 and the data is on an SSD capable of 2GBps, so I assume it should take on the order of a few seconds to read a few gigs. However I am consistently seeing times ten times longer or more.
The best I got was with 4 pvservers reading the Ensight data, with the following timer logs:
PARAVIEW TIMER LOG:
RenderView::Update, 15.7664 seconds
ensight::GatherInformation, 21.3152 seconds <----------- ???
Contour::GatherInformation, 0.469148 seconds
REPRESENTATIVE SERVER TIMER LOG:
RenderView::Update, 15.7663 seconds
vtkPVView::Update, 15.7659 seconds
Execute vtkPGenericEnSightReader id: 8434, 12.0105 seconds
Something seems to be wrong. Am I missing some magic compile flag or another?