Extracts/Exports and Catalyst Python Scripts

In theory, yes. In practice, we encounter this issue with the current implementation. There will be some fine-tuning needed to enable updating extract generators from a Live session in the GUI. We can do that in a following iteration. I don’t think it’s going to be complicated.

+1

Yeah, the view extracts are awkward. I guess you could show them at the same level as sources (same hack we use for fan-in).

That does not bother me. You won’t have these “by-standers” unless you specifically add them, so having these mostly passive pipeline objects shouldn’t be too confusing. And the “prime real-estate” they are taking up is a lot less than an entire new panel. From a GUI footprint perspective, integrating with the Pipeline Browser is the smallest impact we can do.

I don’t see the issue of having a sink in the pipeline that nothing can connect to. It just has no eyeball and all filters (and other extracts) are grayed-out when one is selected. That is pretty consistent with the rest of the GUI.

I don’t think the comparison between extracts and representations is a good one. Representations are the glue between a pipeline object and a view. They are an implementation detail that user’s don’t have to deal with (unless they dive into scripting). I don’t think the GUI considerations between the two match very well.

Good points. I think I am warming up to the idea. That does gracefully solve the issue mentioned earlier. Having a separate Extract Generators menu next to Sources, Filters also makes it easily discoverable rather than being hidden under some panel.

Okay, I’m sold! I’ll need to revisit some of the implementation but shouldn’t be too complicated.

Another added bonus: extract generators can be added to the quick search box already used for sources and filters.

1 Like

If Catalyst’s Live we’re able to finally handle screenshots, that would be incredible but that may be asking too much.

Based on @Kenneth_Moreland’s suggestion, we should be able to easily update screenshot params from Live. In addition, we can just as easily add a “generate extracts now” button that works in “live” mode too where you can generate all (or selected) extracts for the current timestep irrespective of the settings on the extract generators.

1 Like

Here’s what the Pipeline Browser looks like with extract generators. Different icons help distinguish image vs data extract generators.

image

1 Like

I like this – it seems very clear to me. What kind of things will I see when I click on the PNG1 or Cinema1 object in the Pipeline Browser?

Here’s a non-advanced view of the properties panel:

The advanced view has other params shown in the Save screenshot dialog.

Cinema will have Cinema related param, but I haven’t added those yet.

I was actually thinking related to the views more than the Properties panel. I supposed there isn’t any state with the image output though so maybe it’s not relevant, at least for the exporter. For a Live connection though would it just deliver the images that are being produced or would it actually do something with the render view?

Now that I think about it, should there be the concept of locking a view based on the screenshot output? That may be too much for users but I’m concerned about setting the screenshot output, then modifying the view and finally exporting a script and the user not realizing that the screenshot isn’t what they intended. Maybe a thumbnail of the screenshot as a last second check during the export step could alleviate that? Just spit-balling here really fast without thinking this through too much, and not considering the work involved…

For a Live connection though would it just deliver the images that are being produced or would it actually do something with the render view?

In this pass, don’t think I’ll add support for viewing extracts generated in a Live session; only add ability to edit params for the extract generators in a Live session. We could indeed think of ways of supporting that, but that probably warrants its own discussion thread.

Now that I think about it, should there be the concept of locking a view based on the screenshot output?

+1…something that integrates with Preview mode would be good.

Maybe a thumbnail of the screenshot as a last second check during the export step could alleviate that?

One can use Generate Extracts Now or save Catalyst script and run using pvbatch to confirm that the extracts generated are as expected before doing an in situ run.

First pass of this has now been merged in master.

1 Like

Finally got around to reading this. Feels like reading a Michener book. +1. Well done.

sorry! this thread got long very quickly!

I think what we need next is a set of short tutorials to get folks started with these extract generators and catalyst python script changes.

I like these changes a lot! This looks like a complete redo that’s probably been overdue for some time. Some ways to possibly improve:

  1. Would it be possible to add in a hover to the Extracts Generators->Data->* ? I know what many of them are but not sure what things like VTPD is. I see that it’s greyed out for my specific output so maybe it doesn’t matter though.
  2. Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.
  3. For the example it would be nice to have something written to disk as well. I added a dataset writer manually and probably didn’t do it the proper new way but others may have trouble doing that.
  4. When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__. Is that a bug?
  5. If there are more than one channels, is there something to handle this for the -c option?
  6. It would be great to have something available like gridwriter.py and allinputsgridwriter.py that could write out the full grid if the default input channel is used and for any number of channels are used, respectively. I think a number of people have used these in the past to help them get going in the Catalyst workflow.
  7. What is Frequency for in the Generate Catalyst Script Options window under Global Options?

I very briefly skimmed the user guide updates so apologies if I missed something in there. The example did work nicely for me.

Firstly, thanks for trying it out, Andy! Very much appreciated!

Would it be possible to add in a hover to the Extracts Generators->Data->* ?

Indeed. Reported here

Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.

Good point. I wonder if we should really not burden the user with which flavor of writer to use, instead have single entry for VTK-XML (or something more readable) and then create the *.vt??? variant based on the input dataset type. Thoughts?

For the example it would be nice to have something written to disk as well.

Sure thing. I’m assuming you’re referring to the examples here. Reported here.

When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__ . Is that a bug?

I do not follow. Can you elaborate please?

If there are more than one channels, is there something to handle this for the -c option?

-c option is for the wavelet_driver, which only produced 1 channel. Are you suggesting we extend that miniapp to produce multiple channels for demostration purposes? Sure, I don’t have any objections there. Shouldn’t be too hard.

It would be great to have something available like gridwriter.py and allinputsgridwriter.py

Makes sense. Reported here.

What is Frequency for in the Generate Catalyst Script Options window under Global Options?

It is same as every “n-th”. Global options are checked before each extract generator specific trigger options (or live specific trigger options) are checked. Frequency is probably not the right term, but that’s what we’ve used previously so for the lack of a clearer alternative, I left that unchanged.

Thanks!

In that case, your writer will be activated when timestep_index % 5 == 0 && timestep_index % 3 == 0, thus for the 0th, 15th, 30th, … timestep (or cycle).

I guess this adds another question – should the Trigger option be Time or Time Step/Cycle?

Currently, we only have timestep/cycle based triggers. However, the trigger infrastructure is extensible. We should be able to support other types of triggers in the future, even Python-based ones.

I’m just thinking that the Trigger option should be named Time Step instead of Time to be more accurate. In the future I think the first one I’d like to see added is where it’s actually based on simulation time instead of simulation time step. As an example, as close to every half a second in simulation time so if you’re trying to make an animation from images that are dumped in situ that it’s doable, especially for simulations that have variable time step lengths.