Efficiently Visualizing Trajectory Data from NetCDF in ParaView

[_Marco]

Hi all,

I’m working with a NetCDF file that contains trajectory data — x, y, z as a function of time.

Problem Description:

1. The NetCDF file uses custom dimensions (not standard lat, lon, time, level), and ParaView raises a dimensionality error on import.

2. The file is already highly compressed and optimized for storage.

3. It includes proper metadata and is in a format widely supported by the scientific community.

What I’ve Tried: I attempted to convert each time step into a .vtu file, and list them using a .pvd file to represent the time series. However, this approach is inefficient because each .vtu file must contain all trajectory data up to time t. This leads to t files, each with t columns — a size that grows as t², which is clearly unsustainable.

Proposed Solution: I’d prefer to write a filter or macro that reads the NetCDF file once and, as ParaView steps through time, extracts and visualizes only the relevant position data (x, y, z) for each time step. This would be far more efficient.

Question: Any suggestions on how to approach this? Are there existing filters, plugins, or similar examples that I could adapt?

Thanks in advance!

[danlipsa (Dan Lipsa (Kitware))]

1. It includes proper metadata and is in a format widely supported by the scientific community.

What is the standard used for the file? Can you share your file?

[_Marco]

Hi Dan!

Thanks for your feedback.

Not sure what you mean by “standard”. It is a normal netCDF4 file, following CF conventions (“CF-1.6, IMOS-1.3” ).

The original file has 1.5 million points (1.5 M timesteps, 140 MB), so I have sliced the first 50, for portability (now 300 KB). The structure remains of course unchanged. I am attaching it to this message.

Grateful for your help.

Cheers,
Marco
IMOS_ANFOG_BCEOPSTUV_20111121T051804Z_SL239_FV01_timeseries_END-20111212T211841Z_subset_first50.nc (307.6 KB)

[danlipsa (Dan Lipsa (Kitware))]

Looking at your data, while lat, lon are defined, you don’t have any variables that depend on lat,lon. This means you don’t have any dimensional data that ParaView can show. Maybe those variables did not get saved.

[_Marco]

Dear Dan,

Yes - you are right

@danlipsa

you don’t have any variables that depend on lat,lon

That was my point when opening this post. Lat, lon (as well as depth, salinity, temperature an many more) are indeed there, just they are a function of time.
x = x(t), y = y(t), z = z(t) with t = [0, N]
That is a vector of N samples.
The topology, in this case, is non-standard (not, for instance temperature = f (lat, lon, time)) and need, therefore, special treatment.

@danlipsa

Maybe those variables did not get saved

Negative! I am absolutely certain variables are there and populated with data. What follows is a ncdump of my netcd file (the ony I attached to this chat). You can see everything is right where it should be.

netcdf IMOS_ANFOG_BCEOPSTUV_20111121T051804Z_SL239_FV01_timeseries_END-20111212T211841Z_subset_first50 {
dimensions:
        TIME = UNLIMITED ; // (50 currently)
        string1 = 1 ;

	SENSOR1:sensor_parameters = "TEMP, CNDC, PRES, PSAL" ;
	double TEMP(TIME) ;
		TEMP:_FillValue = 99999. ;
		TEMP:standard_name = "sea_water_temperature" ;
		TEMP:long_name = "sea_water_temperature" ;
		TEMP:units = "Celsius" ;
		TEMP:valid_min = -2.5 ;
		TEMP:valid_max = 40. ;
		TEMP:ancillary_variables = "TEMP_quality_control" ;
		TEMP:observation_type = "measured" ;
		TEMP:quality_control_set = 1b ;
		TEMP:comment = "Data have been corrected for sensor time response. Uncorrected data are available in the FV00 file associated with this mission. " ;
		TEMP:coordinates = "TIME LATITUDE LONGITUDE DEPTH" ;




	LATITUDE_quality_control:coordinates = "LATITUDE LONGITUDE DEPTH" ;
	double LONGITUDE(TIME) ;
		LONGITUDE:_FillValue = 99999. ;
		LONGITUDE:standard_name = "longitude" ;
		LONGITUDE:long_name = "longitude" ;
		LONGITUDE:units = "degrees_east" ;
		LONGITUDE:valid_min = -180. ;
		LONGITUDE:valid_max = 180. ;
		LONGITUDE:comment = "obtained from GPS fixes" ;
		LONGITUDE:reference_datum = "geographical coordinates, WGS84 projection" ;
		LONGITUDE:ancillary_variables = "LONGITUDE_quality_control" ;
		LONGITUDE:observation_type = "measured" ;
		LONGITUDE:quality_control_set = 1b ;
		LONGITUDE:axis = "X" ;

The whole point in this is that instead of the conventional netcdf reader with conventional topology, a new strategy should be developed (easy part) and implemented (head-scratching part):

@_Marco

Proposed Solution: I’d prefer to write a filter or macro that reads the NetCDF file once and, as ParaView steps through time, extracts and visualizes only the relevant position data (x, y, z) for each time step. This would be far more efficient.

PV only knows standard topologies (with netcdf files). But there is a lot more than that.

@_Marco

Question: Any suggestions on how to approach this? Are there existing filters, plugins, or similar examples that I could adapt?

Thoughts?

Thank you very much again for your effort.

[danlipsa (Dan Lipsa (Kitware))]

So, are you expecting only one TEMP value per time step? That would result in a point in ParaView. The NetCDF CF reader knows to do this kind of indirection and it gets only one point when choosing () as Dimensions, but for some reason the point ParaView shows is undefined. Seems to be a bug in the reader.

[_Marco]

No, not really. I would expect all points up to the considered timesteps.

So, when animating I would ideally loop through time. I would find myself at, say, time t0: here I would see appearing a trail (temperature, salinity, pressure, etc…), where the head is positioned at location x(t0), y(t0), z(t0). The trail would be t0 samples long, and have all the values from 0 to t0.

@_Marco

What I’ve Tried: I attempted to convert each time step into a .vtu file, and list them using a .pvd file to represent the time series. However, this approach is inefficient because each .vtu file must contain all trajectory data up to time t. This leads to t files, each with t columns — a size that grows as t², which is clearly unsustainable.

The above is not efficient and I would hope to exploit the already existing netcdf, without having to reinvent the wheel.

Ideally, some kind of filter or macro should loop through the only available dimension (time) and take care of:

1. Positioning my point in space (x, y, z at time t0) and
2. coloring it according to the active variables selected.

Extra bonus:

• include in the code some function that computes the cilumulative distance from origin;
• add the possibility of having time varying transects (that is including vertical dimension).

My skills in programmable filters aren’t up the challenge, so i am reaching out here to seek for advice or a good start where I can build upon.

Thanks.

[danlipsa (Dan Lipsa (Kitware))]

Something like this? But we have to fix the bug so that we do see one point for each timestep.

[_Marco]

YES!!

That looks very good.

Now, how can I help and contribute to make that work, and have this feature implemented as a standard capability?

Thank you very much for effort and patience so far.

Let me know what I can do to help (if at all possible)

Marco

[danlipsa (Dan Lipsa (Kitware))]

This is the ParaView state file for the previous image which uses can.ex2 example dataset.
path_along_time.pvsm (598.5 KB)
What is needed next is to figure out why vtkNetCDFCFReader sets the point it reads from the file undefined. In increasing order of difficulty: you could file a bug report for this issue, compile paraview in debug mode and look with a debugger to see why the reader does not set the point correctly.