Custom CSV Format with Two Different Field Lengths for Import

Greetings,

I have CSV files with a 7 row pattern which constitutes one point in time. The first row is 6 fields. The second to the seventh row are 10 fields. This can go one for 10’s of megabytes.

How can I teach Paraview this format for ingesting the data?

2021-06-09 00:00:00.442,-1,-1,40AD,43,48
2021-06-09 00:00:00.442,1,0,1010,0,42,65535,37,14709,21142
2021-06-09 00:00:00.442,1,1,237,1948,44,65535,36,14913,21142
2021-06-09 00:00:00.442,2,0,915,0,42,65535,36,14719,21139
2021-06-09 00:00:00.442,2,1,242,1978,43,65535,38,14910,21139
2021-06-09 00:00:00.442,3,0,422,0,45,65535,37,14699,13519
2021-06-09 00:00:00.442,3,1,294,1960,46,65535,36,14898,13519
2021-06-09 00:00:10.632,-1,-1,40AD,43,49
2021-06-09 00:00:10.632,1,0,550,1952,42,65535,37,14802,21142
2021-06-09 00:00:10.632,1,1,279,0,44,65535,36,14815,21142
2021-06-09 00:00:10.632,2,0,657,1970,42,65535,36,14807,21139
2021-06-09 00:00:10.632,2,1,262,0,43,65535,38,14817,21139
2021-06-09 00:00:10.632,3,0,829,0,45,65535,37,14693,13519
2021-06-09 00:00:10.632,3,1,234,1960,46,65535,36,14905,13519
2021-06-09 00:00:20.187,-1,-1,40AD,43,49
2021-06-09 00:00:20.187,1,0,297,1952,42,65535,37,14811,21142
2021-06-09 00:00:20.187,1,1,489,0,44,65535,36,14808,21142
2021-06-09 00:00:20.187,2,0,308,1970,42,65535,36,14821,21139
2021-06-09 00:00:20.187,2,1,456,0,43,65535,38,14807,21139
2021-06-09 00:00:20.187,3,0,798,1963,45,65535,37,14787,13519
2021-06-09 00:00:20.187,3,1,242,0,46,65535,36,14815,13519

Currently, Paraview only reads up to Field 5 (sixth field) and then ignores the remaining columns.

Cheers,
Joe

No you can’t unless you manually (using sed or another tools) add zeroes or Nans in the missing collumns.

Modifying the csv reader to support that should be doable, as well as writing a simple ProgrammableSource to read the data yourself.

1 Like

Mathieu,

Thank you for writing.

I found the CSV Reader. There appears to be a C++ reference and Python plugin:
https://gitlab.kitware.com/paraview/paraview/-/blob/master/VTKExtensions/IOGeneral/vtkEnsembleDataReader.cxx#L211

https://gitlab.kitware.com/paraview/paraview/-/blob/master/Examples/Plugins/PythonAlgorithm/PythonAlgorithmExamples.py#L132

And the ProgrammableSource is a python script output:
https://kitware.github.io/paraview-docs/latest/python/paraview.simple.ProgrammableSource.html

Choices.

I’m not sure that your file fits CSV file format. In your case, one point in time is defined by several line with variable length. CSV expects one line per entry with fixed row count.

To stick with the CSV file format I would recommend to merge the 7 rows pattern to one row in order to get the 1 line = 1 entry contract.

Your first point defined by this:

A1,B1,C1
D1,E1
F1,G1
A2,B2,C2
D2,E2
F2,G2

should become:

A1,B1,C1,D1,E1,F1,G1
A2,B2,C2,D2,E2,F2,G2

Best,