Missing methods in generated Catalyst v2 pipeline or wrong usage ?

In Situ Support
1

Hello,

I tried to test the Catalyst changes in ParaView master (recompiled yesterday), but when I extract a catalyst script using File/Save Catalyst state, the file I obtain does not contain the methods mentioned in commit af47a138 (catalyst_initialize, catalyst_finalize, catalyst_finalize), and the script seems to call an EnSight reader (used on the initial data) rather than an input from a CataLyst adapter. Basically, it is the same as a regular Python trace file except for the __main section.

Am I missing something or not using the correct method/tool to generate the Catalyst script ?

Best regards,

Yvan

2

Just curious – are you using the new Catalyst API (based on Conduit) or the legacy Catalyst API?

3

Here’s a post that provides an overview of these changes.

Here’s a document with the details of the scripts generated by ParaView for Catalyst starting with 5.9.

In summary,

  • ParaView UI now exports Catalyst scripts that rely on Extractors to define which extracts to generate.
  • On the input side, i.e. which pipeline object in the Python script gets replaced by simulation input, we simply use the registrationName attribute. If the name matches a named channel in Catalyst, it gets replaced by the producer for that channel. This is the reason why you still see the EnSightReader in the state, but it will get replaced with executing in side the Catalyst environment with the proxy for a TrivialProducer if the name matches the channel name. This trick also makes it so that the generated script is still fully functional and usable with pvbatch. The hope is that that will make it easier to test these scripts.

The documentation here describes this, so do let me know if it’s not clear and we can improve the docs.

1 Like
4

I think the generated Catalyst script is still reading from the file instead of generating from the adaptor. I’m using the CxxImageDataExampleV2 example and if I change the reader to be something like:

dataset0098vtpd = XMLPartitionedDatasetReader(registrationName='dataset-0098.vtpd', FileName=['/home/acbauer/Code/ParaView/JUNK/Catalyst2/CxxImageDataExample/Testing/Temporary/dataset-0098.vtpdGARBAGE'])

and run the script it fails but if I pass in a proper file name (e.g. take out GARBAGE it works). Also, if I run my script through that example when it does work none of the values are changing so it’s probably just getting the same data from the file instead of actually from the example driver.

5

Hello,

Yes, this seems to be the issue. Defining extracts under the ParaView GUI works fine, but the exported Catalyst script (using the “File/Save Catalyst state” menu entry) does not seem to match the documentation (i.e. does not contain catalyst_initialize/catalyst_execute/catalyst_finalize). Maybe this is not implemented yet ?

As for my adaptor (here https://github.com/code-saturne/code_saturne/blob/master/src/fvm/fvm_to_catalyst.cxx), I only did minor updates so as to also use the V2 pipeline, but still go through VTK structures, and do not do a direct conduit description. I have barely scratched at the conduit documentation, and have not yet started any work on this.

Best regards,

Yvan

6

It could just be that for the Catalyst V2 API that the registrationName, channel is incorrect in the generated script because the channel name isn’t included in the output file. This may just be for the VTPD writer or for the CxxImageDataExampleV2. Not sure where the failure is right now.

7

Note that I did not try running the case under Catalyst yet, so it hasn’t “failed” so far.

Simply, checking the form of the file generated by ParaView, the input reader is not automatically replaced with an “in-situ” input. I could certainly edit it and add functions so it matches the new pipeline, but was hoping the file generated from the ParaView GUI would be usable “as is” (as I already have trouble pushing more of our users to use Catalyst, so hope generating scripts will remain as easy as before).

I might simply be too impatient and be testing “work in progress”, but since the behavior seemed surprising, I preferred to report this here.

8

Okay I am going to try to respond here, but we’re having two different conversations at the same time and it’s Friday morning and the coffee hasn’t kicked in yet :).

issue #1:

So, I am confused why this is expected to work with CxxImageDataExampleV2. CxxImageDataExampleV2 names the channel grid. This can be confirmed by looking at the code here.

The registrationName in the Script should be changed to grid or before saving the Catalyst state make sure you rename the source in the Pipeline browser to match the channel name.

Maybe the issue you’re actually reporting is that when the dataset saved out from Catalyst is loaded back in ParaView, it does not automatically get renamed in the pipeline browser to match the Channel name. Yup, that’s a bug. I’ve reported it here.

Update: a fix is available here.

issue #2:

See the documentation here. Specifically, the following extract:

In ParaView 5.9, we introduced a new pipeline object called Extractors . Extractors are items in the visualization pipeline that can save data or images per timestep. Extractors make it largely unnecessary to have any custom code to execute per iteration since one can simply use extractors to save out image extracts from views or data extracts from filters and other data producers.

Thus, when using Extractors, custom code for catalyst_execute, catalyst_initialize etc. is not even needed and hence File > Save Catalyst State does not add it.

Glad you did! I want to make sure we make the documentation as clear as possible. So if you notice any discrepancies, please don’t hesitate. Early adopters are always welcome. I know it can be very painstaking but truly appreciated! Thank you!

9

Ok thanks for the additional explanations.

Issue #2 explains why the generated script did not have the form I expected.

Regarding Issue #1, if I understand correctly, when running under Catalyst, the registrationName will be used instead of the fileName ? If I change the name under ParaView before saving the Catalyst state, it is renamed fine. So in this case everything seems fine, and I can move to the next stage of testing (actually loading the script in the running code).

I guess I will need to upgrade to the conduit API first, as I may have missed more aspects of the new pipeline.

A side question (which was actually probably at the origin of my confusion) is that using ParaView 5.9, it is not possible to generate Catalyst V1 scripts anymore. Without going through a “deprecated” phase (or a very short one at that). Or it might still be possible from Python ? Just to give me the time to adapt my code.

Best regards,

Yvan

10

This is not necessary. This is totally independent change. The Conduit+Catalyst Adaptor changes are at alpha stage right now, so I’d only recommend doing that for prototyping and prepping for the future.

That is correct. ParaView can no longer export Python scripts that work with vtkCPPythonScriptPipeline. You’ll need to use older versions of ParaView for that. Teaching a previously instrumented simulation to work with Python V2 scripts is easy though. Just instantiate vtkCPPythonScriptV2Pipeline instead. Merged in this morning are changes to vtkCPPythonPipeline that add static methods to detect the version of a Python script using some heuristics.

11

Hello,

Thanks for the info. I tested with a V2 script (loading the pipeline with vtkCPPythonScriptV2Pipeline) but have an issue I do not have with V1 scripts.

I get a an exception (SIGSEGV) under vtkCPProcessor.cxx (line 234 as of commit 86fbe2b215b1cba6ca0d223635741df7d725bc21)

│   230           int doCoProcessing = 0;                                                                       │
│   231           for (vtkCPProcessorInternals::PipelineListIterator iter = this->Internal->Pipelines.begin();  │
│   232                iter != this->Internal->Pipelines.end(); iter++)                                         │
│   233           {                                                                                             │
│  >234             if (iter->GetPointer()->RequestDataDescription(dataDescription))                            │
│   235             {                                                                                           │
│   236               doCoProcessing = 1;                                                                       │
│   237             }                                                                                           │
│   238           }                                                                                             │
│   239           return doCoProcessing;

Where under gdb,

call dataDescription->GetNumberOfInputDescriptions()

returns 1, but iter seems wrong:

(gdb) p iter
$23 = {<vtkSmartPointerBase> = {Object = 0x555555fd2b00}, <No data fields>}

I guess the next step is to check what these values are in a working (V1 pipeline) script in my application.

If I check the dataDescription object, I have similar contents both for the V1 and V2 pipelines (here from V2:

p *((vtkCPInputDataDescription *) 0x555556fde270)
$27 = {<vtkObject> = {<vtkObjectBase> = {
_vptr.vtkObjectBase = 0x7ffff4d75720 <vtable for vtkCPInputDataDescription+16>,
ReferenceCount = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = 1}, <No data fields>},
WeakPointers = 0x0, IsInMemkind = false}, Debug = false, MTime = {ModifiedTime = 286509},
SubjectHelper = 0x0}, AllFields = false, GenerateMesh = false, Grid = 0x0, TemporalCache = 0x0,
Internals = 0x55555725b360, WholeExtent = {0, -1, 0, -1, 0, -1}}

Running under Valgrind, I get a bit more information:

==926622== Invalid read of size 8
==926622==    at 0x7AA7E17: vtkCPProcessor::RequestDataDescription(vtkCPDataDescription*) (vtkCPProcessor.cxx:234)
==926622==    by 0x4966CE0: fvm_to_catalyst_flush (fvm_to_catalyst.cxx:1876)
==926622==    by 0x494401B: fvm_writer_flush (fvm_writer.c:1562)
==926622==    by 0x4B2933B: cs_post_time_step_end (cs_post.c:6934)
==926622==    by 0x4B2941B: cs_post_write_vars (cs_post.c:6980)
==926622==    by 0x4B2ACB0: pstvar_ (cs_post_default.c:398)
==926622==    by 0x4986757: caltri_ (caltri.f90:1096)
==926622==    by 0x484D0C7: _run (cs_solver.c:440)
==926622==    by 0x484D3B5: main (cs_solver.c:677)
==926622==  Address 0x203d6860 is 0 bytes inside a block of size 64 free'd
==926622==    at 0x483BEAB: operator delete(void*) (vg_replace_malloc.c:584)
==926622==    by 0x49673D1: vtkNew<vtkCPPythonScriptV2Pipeline>::Reset() (vtkNew.h:127)
==926622==    by 0x49672A1: vtkNew<vtkCPPythonScriptV2Pipeline>::~vtkNew() (vtkNew.h:119)
==926622==    by 0x4963A97: _add_v2_pipeline(char const*) (fvm_to_catalyst.cxx:345)
==926622==    by 0x4963BDE: _add_script(char const*) (fvm_to_catalyst.cxx:423)
==926622==    by 0x4963F52: _add_dir_scripts(char const*) (fvm_to_catalyst.cxx:496)
==926622==    by 0x49662C4: fvm_to_catalyst_init_writer (fvm_to_catalyst.cxx:1530)
==926622==    by 0x494298A: _format_writer_init (fvm_writer.c:715)
==926622==    by 0x494376F: fvm_writer_init (fvm_writer.c:1211)
==926622==    by 0x4B1DDA8: _init_writer (cs_post.c:590)
==926622==    by 0x4B20D28: _cs_post_write_mesh (cs_post.c:2207)
==926622==    by 0x4B2696C: cs_post_write_meshes (cs_post.c:5454)
==926622==  Block was alloc'd at
==926622==    at 0x483ADEF: operator new(unsigned long) (vg_replace_malloc.c:342)
==926622==    by 0x7A8F2C0: vtkCPPythonScriptV2Pipeline::New() (vtkCPPythonScriptV2Pipeline.cxx:21)
==926622==    by 0x496727A: vtkNew<vtkCPPythonScriptV2Pipeline>::vtkNew() (vtkNew.h:89)
==926622==    by 0x49638CC: _add_v2_pipeline(char const*) (fvm_to_catalyst.cxx:345)
==926622==    by 0x4963BDE: _add_script(char const*) (fvm_to_catalyst.cxx:423)
==926622==    by 0x4963F52: _add_dir_scripts(char const*) (fvm_to_catalyst.cxx:496)
==926622==    by 0x49662C4: fvm_to_catalyst_init_writer (fvm_to_catalyst.cxx:1530)
==926622==    by 0x494298A: _format_writer_init (fvm_writer.c:715)
==926622==    by 0x494376F: fvm_writer_init (fvm_writer.c:1211)
==926622==    by 0x4B1DDA8: _init_writer (cs_post.c:590)
==926622==    by 0x4B20D28: _cs_post_write_mesh (cs_post.c:2207)
==926622==    by 0x4B2696C: cs_post_write_meshes (cs_post.c:5454)

Note thet with the V2 pipeline, under gdb, I often get messages similar to this:

warning: Corrupted shared library list: 0x5555556a3260 != 0x7fffe23c48a0

Which I did not observe with the V1 pipeline (same build, but I ran the V2 pipeline more often under the debugger). To run under gdb, I linked ParaView statically. I don’t know if this might be the root cause of the issue, but seeing that there is a specific V2 pipeline path in the Valgrinf trace, I don’t know if the issue has more chances of hiding in the build process, or in the source code…

Any help on this is welcome.

Best regards,

Yvan

12

Mind sharing the code where you are creating the vtkCPPythonScriptV2Pipeline instance and registering it? I suspect there’s an accidental double-delete happening there.

13

Hello,

Yes, this seems to be it. I just uploaded the working version of my code (an update relative to the master branch of code_saturne, mirrored at https://github.com/code-saturne/code_saturne.

Commenting line 366 (as I just did in the attached version), the crash disapears and the code seems to work fine.

Just to understand, I had a similar pipeline->Delete() for the V1 pipeline, which seemed to do fine, and which I had taken from examples if I remember correctly (assuming this was used to decrement a reference counter, which was incremented by the previous AddPipline).

Should I also remove the Delete() for the V1 pipeline (line 452), or leave it as is ? (I try to have “clean code” as regards even minor memory leaks, so that instrumentation provided by Valgrind or clang/gcc Sanitizers remains as usable as possible/not polluted by small leaks, so I may be a bit zealous with freeing memory).

fvm_to_catalyst.cxx (52.3 KB)

Thanks in any case. I now have things working again, and should be able to remove the 1st step we have in our code_saturne + Catalyst instructions [edit: not sure about that actually, but at worst we keep this stage the same] , which is the trick users forget easily. Having almost the same script for saving state and Catalyst is a nice improvement.

I’ll probably try to upgrade to conduit sometime in 2021, but at least now our code seems ready for ParaView-5.9.

Best regards,

Yvan

14

Here’s relevant extract from the fvm_to_catalyst.cxx:

/// 
vtkNew<vtkCPPythonScriptV2Pipeline> pipeline;
...
  _processor->AddPipeline(pipeline);
 pipeline->Delete();

////
vtkCPPythonScriptPipeline  *pipeline = vtkCPPythonScriptPipeline::New();
....
  _processor->AddPipeline(pipeline);
  pipeline->Delete();

The difference is vtkNew<...> vs ...::New. The former, i.e. vtkNew<..> creates a scoped variable that is release the VTK object reference automatically when the variable goes out of scope i.e. similar to calling ...->Delete(). The latter, i.e. ::New doesn’t go this clean up and hence you have to manually call delete.

To avoid the confusion, I’d recommend updating both locations to use vtkNew<...> and remove all explicit ->Delete() calls.

15

Thanks for the explanation and recommendation.

Done.

Best regards,