Information is out of date when using get_execute_params() inside a python trigger for extractors

In Situ Support
1

A few years ago, we implemented python triggers to ParaView Catalyst extractors. Extractors can run a piece of code that returns True or False to determine whether they should be triggered. We also implemented paraview.simple.catalyst.get_execute_params(), which reads the state/parameters node in the Conduit tree.

These two features were intended to be used together: The simulation solver passes trigger information through the parameters node, and the extractor trigger script fetches that information using get_execute_params().

Here’s the thing though: When the extractor evaluates the trigger script, the pipeline is of of date. That’s actually the whole point, since we are still trying to figure out whether we need to update. But if the pipeline is out of date, then ultimately the Conduit source is out of date and get_execute_params() fetches the previous parameters node, not the current one. The work around is to force an UpdatePipeline() in catalyst_execute(info) on the Conduit source.

Is this an issue? As in, did we miss this during the implementation? Or, are we happy with this workaround? I only just realize I’ve been using these two features incorrectly for the past two years or so. It’s a pretty sneaky subetlty.

By the way, the issue the combination of these features is trying to solve is that there is no way to call catalyst_execute() form the simulation code for only some of the scripts or pipelines. catalyst_execute() always runs all the scripts registered in catalyst_initialize().

I’m curious on the community’s thoughts on this.

Alexandre

2

FYI @Louis_Gombert @Francois_Mazen @nicolas.vuaille

3

I have what I think is a similar need for Catalyst. That is, before executing Catalyst for a given timestep I want to be able to check with the script what is needed that time step. Things I’m interested in are figuring out are if anything in the Catalyst scripts will actually get computed that time step and if yes then what channels and potentially what fields in each of those channels. This way before catalyst_execute() is called from the adaptor for a given time step I know what information (e.g. channels, fields, etc.) need to be constructed by the adaptor and passed in to catalyst_execute. Is this similar to the functionality that you’re thinking about? This type of functionality was available in Catalyst V1 but not yet in the V2 API.

4

My need is the other way around. My adaptor knows exactly what needs to be done and will construct only a few channels. However, I have no way to tell ParaView Catalyst which pipeline to run. It will run all the pipelines even, though some pipeline don’t make sense for the channels the adaptor sent. ParaView Catalyst always runs SaveAndExtract on all pipelines, so the only way for this to end up not triggering a pipeline is to disable all extractors on that pipeline.

It would be much easier if we could pass which pipelines need to be executed when the adaptor calls catalyst_execute().

5

Ah, that I think is available in ParaView Catalyst V2. If you do something like:

      node["catalyst/scripts/filename1"].set_string(catalyst_script1);
      node["catalyst/scripts/filename2"].set_string(catalyst_script2);

During your catalyst_initialize() step and then the following when calling catalyst_execute() from the adaptor:

  node["catalyst/state/pipelines/0"] = "filename1";

that should only execute the extractors specified in catalyst_script1. I may be slightly wrong with the syntax but that’s the general idea.

6

Hum, I did not know that. I’ll check it out, thanks.

7

Andy,
I confirmed the syntax you posted works, thanks!

@Christos_Tsolakis , I didn’t find this feature in the documentation or the examples. If you ever have time to add it I think it would be helpful to others.

8

Thanks for the detailed report — after reviewing the implementation, the behavior you’re seeing is due to ordering, not UpdatePipeline().

In Catalyst V2, execution parameters are copied from the adaptor-provided params vector into an internal snapshot (ParametersList) inside vtkCPPythonScriptV2Helper::CatalystExecute(…). That refresh happens after IsActivated() succeeds.

However, extractor trigger evaluation occurs during IsActivated(). So if a trigger script calls get_execute_params(), it will see whatever snapshot was last populated, which may correspond to a previous activation.

Summary:

UpdatePipeline() does not refresh execution parameters.

It does not regenerate the Conduit node.

It only updates VTK pipeline objects.

So forcing UpdatePipeline() is not the real fix; it only appears to help if it changes activation behavior indirectly. UpdatePipeline() only affects activation if the trigger logic depends on pipeline data. It does not affect execution parameter visibility.

The core issue is that parameter refresh is currently gated behind activation. A minimal improvement would be to refresh the execution parameter snapshot at the beginning of CatalystExecute(), before IsActivated() is evaluated. That would allow extractor triggers to see the current timestep’s parameters without lag.

I’d appreciate your thoughts @Andy_Bauer @Alexandre_Minot @berkgeveci or anyone else with a stake. Its an easy fix but has some consequences if folks have built in workarounds already.

9

I think this should be fixed. I actually won’t need the fix as I am now using Andy’s much simpler solution, but there’s no way a user would know what you just explained.

10

Thanks for the feedback Alexandre, I don’t know all of the history but it sounds like Python triggers could have been intended to handle cases where the adaptor doesn’t have enough information to decide what to enable, i.e. results from analysis pipelines or queries of that extracted data. Even in that case we should make sure the execution parameters are up to date and if you need pipeline info then there is no choice but to update dependent proxies to satisfy the query. Not much documentation on python triggers:

Python trigger for extractors

Extractors now support a trigger type which allows users to invoke Python scripts to determine if the extractor should be activated or not. This allows for more flexible triggers than the simple time or timestep based triggers supported previously.

but for Andy’s suggestion, this is in the Paraview-5.13.0.md

Catalyst pipeline selection

Support for which Catalyst pipelines to execute has been added. This includes Python script pipelines as well as precompiled pipelines. The Python script pipelines are configured with:


node['catalyst/scripts/aname'] = afile.py

where aname is the name of the pipeline. Precompiled pipelines are configure with:


node["catalyst/pipelines/0/type"] = "io"

node["catalyst/pipelines/0/filename"] = "foo-%04ts.vtpd"

node["catalyst/pipelines/0/channel"] = "input"

where 0 is the name of the pipeline.

Pipelines can be selected by listing them under the catalyst/state/pipelines node such as


node['catalyst/state/pipelines/0'] = 'aname'

Note that the value of the node (aname) is what matters and not the name of the node (0). It is possible to turn off all of the pipelines by making pipelines a leaf node such as:


node['catalyst/state/pipelines'] = 0

If catalyst/state/pipelines does not exist, Catalyst will, by default, execute all pipelines.