Extracts/Exports and Catalyst Python Scripts

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

@nicolas.vuaille

Can we trigger extracts or modify extracts options at runtime ?

In theory, yes. In practice, we encounter this issue with the current implementation. There will be some fine-tuning needed to enable updating extract generators from a Live session in the GUI. We can do that in a following iteration. I don’t think it’s going to be complicated.

[Dave_DeMarle (Dave DeMarle (Intel))]

+1

[Kenneth_Moreland]

@utkarsh.ayachit

1. there’s no way to show extract-generators attached to the views in the Pipeline browser, only those attached to the data-producers.

Yeah, the view extracts are awkward. I guess you could show them at the same level as sources (same hack we use for fan-in).

@utkarsh.ayachit

2. extract generators don’t participate at all in your standard post-processing visualization workflow: open data, applying filters, interacting etc etc. So, for most of your interactive session they’ll be just sitting there taking up prime real-estate in the Pipeline browser. In an interactive session they do come into action when you hit the Generate Extracts Now button, but until then they are just by-standers.

That does not bother me. You won’t have these “by-standers” unless you specifically add them, so having these mostly passive pipeline objects shouldn’t be too confusing. And the “prime real-estate” they are taking up is a lot less than an entire new panel. From a GUI footprint perspective, integrating with the Pipeline Browser is the smallest impact we can do.

@utkarsh.ayachit

3. extract generators are sinks, so nothing else can be connected to them. If you think about it, representations are sinks too, and they don’t show up in the pipeline browser either (unless we consider the eyeballs). They only show up on the Properties panel. The Extracts Inspector (maybe we should call is Extracts Properties Panel or Extracts Panel ?), in similar vein, should extract generator properties for the active source. We could indeed be showing these on the Properties panel itself, but that complicates an already complex and long panel. It may indeed be useful, however, to add an indicator to the Pipeline Browser , similar to an eyeball but some other icon, to indicate which data-producer has enabled extract-generators configured.

I don’t see the issue of having a sink in the pipeline that nothing can connect to. It just has no eyeball and all filters (and other extracts) are grayed-out when one is selected. That is pretty consistent with the rest of the GUI.

I don’t think the comparison between extracts and representations is a good one. Representations are the glue between a pipeline object and a view. They are an implementation detail that user’s don’t have to deal with (unless they dive into scripting). I don’t think the GUI considerations between the two match very well.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

Good points. I think I am warming up to the idea. That does gracefully solve the issue mentioned earlier. Having a separate Extract Generators menu next to Sources, Filters also makes it easily discoverable rather than being hidden under some panel.

Okay, I’m sold! I’ll need to revisit some of the implementation but shouldn’t be too complicated.

[Kenneth_Moreland]

Another added bonus: extract generators can be added to the quick search box already used for sources and filters.

[Andy_Bauer]

If Catalyst’s Live we’re able to finally handle screenshots, that would be incredible but that may be asking too much.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

Based on @Kenneth_Moreland’s suggestion, we should be able to easily update screenshot params from Live. In addition, we can just as easily add a “generate extracts now” button that works in “live” mode too where you can generate all (or selected) extracts for the current timestep irrespective of the settings on the extract generators.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

Here’s what the Pipeline Browser looks like with extract generators. Different icons help distinguish image vs data extract generators.

[Andy_Bauer]

I like this – it seems very clear to me. What kind of things will I see when I click on the PNG1 or Cinema1 object in the Pipeline Browser?

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

Here’s a non-advanced view of the properties panel:

The advanced view has other params shown in the Save screenshot dialog.

Cinema will have Cinema related param, but I haven’t added those yet.

[Andy_Bauer]

I was actually thinking related to the views more than the Properties panel. I supposed there isn’t any state with the image output though so maybe it’s not relevant, at least for the exporter. For a Live connection though would it just deliver the images that are being produced or would it actually do something with the render view?

Now that I think about it, should there be the concept of locking a view based on the screenshot output? That may be too much for users but I’m concerned about setting the screenshot output, then modifying the view and finally exporting a script and the user not realizing that the screenshot isn’t what they intended. Maybe a thumbnail of the screenshot as a last second check during the export step could alleviate that? Just spit-balling here really fast without thinking this through too much, and not considering the work involved…

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

For a Live connection though would it just deliver the images that are being produced or would it actually do something with the render view?

In this pass, don’t think I’ll add support for viewing extracts generated in a Live session; only add ability to edit params for the extract generators in a Live session. We could indeed think of ways of supporting that, but that probably warrants its own discussion thread.

Now that I think about it, should there be the concept of locking a view based on the screenshot output?

+1…something that integrates with Preview mode would be good.

Maybe a thumbnail of the screenshot as a last second check during the export step could alleviate that?

One can use Generate Extracts Now or save Catalyst script and run using pvbatch to confirm that the extracts generated are as expected before doing an in situ run.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

First pass of this has now been merged in master.

• User guide updates
• Catalyst Python Scripting details

[wascott (Walter Alan Scott)]

Finally got around to reading this. Feels like reading a Michener book. +1. Well done.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

sorry! this thread got long very quickly!

I think what we need next is a set of short tutorials to get folks started with these extract generators and catalyst python script changes.

[Andy_Bauer]

I like these changes a lot! This looks like a complete redo that’s probably been overdue for some time. Some ways to possibly improve:

1. Would it be possible to add in a hover to the Extracts Generators->Data->* ? I know what many of them are but not sure what things like VTPD is. I see that it’s greyed out for my specific output so maybe it doesn’t matter though.
2. Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.
3. For the example it would be nice to have something written to disk as well. I added a dataset writer manually and probably didn’t do it the proper new way but others may have trouble doing that.
4. When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__. Is that a bug?
5. If there are more than one channels, is there something to handle this for the -c option?
6. It would be great to have something available like gridwriter.py and allinputsgridwriter.py that could write out the full grid if the default input channel is used and for any number of channels are used, respectively. I think a number of people have used these in the past to help them get going in the Catalyst workflow.
7. What is Frequency for in the Generate Catalyst Script Options window under Global Options?

I very briefly skimmed the user guide updates so apologies if I missed something in there. The example did work nicely for me.

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

Firstly, thanks for trying it out, Andy! Very much appreciated!

Would it be possible to add in a hover to the Extracts Generators->Data->* ?

Indeed. Reported here

Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.

Good point. I wonder if we should really not burden the user with which flavor of writer to use, instead have single entry for VTK-XML (or something more readable) and then create the *.vt??? variant based on the input dataset type. Thoughts?

For the example it would be nice to have something written to disk as well.

Sure thing. I’m assuming you’re referring to the examples here. Reported here.

When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__ . Is that a bug?

I do not follow. Can you elaborate please?

If there are more than one channels, is there something to handle this for the -c option?

-c option is for the wavelet_driver, which only produced 1 channel. Are you suggesting we extend that miniapp to produce multiple channels for demostration purposes? Sure, I don’t have any objections there. Shouldn’t be too hard.

It would be great to have something available like gridwriter.py and allinputsgridwriter.py

Makes sense. Reported here.

What is Frequency for in the Generate Catalyst Script Options window under Global Options?

It is same as every “n-th”. Global options are checked before each extract generator specific trigger options (or live specific trigger options) are checked. Frequency is probably not the right term, but that’s what we’ve used previously so for the lack of a clearer alternative, I left that unchanged.

[Andy_Bauer]

Thanks!

@utkarsh.ayachit

Firstly, thanks for trying it out, Andy! Very much appreciated!

Would it be possible to add in a hover to the Extracts Generators->Data->* ?

Indeed. Reported here

Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.

Good point. I wonder if we should really not burden the user with which flavor of writer to use, instead have single entry for VTK-XML (or something more readable) and then create the *.vt??? variant based on the input dataset type. Thoughts?

Not a bad idea to just say Parallel VTK XML output and create the .pvt? variant, if it can be done in a non-confusing way. We’ve been doing it for so long though with .pvti, .pvtu, .vtm, etc. though that maybe we’ve trained the users too much in that way of thinking. I’ll leave it up to you though.

For the example it would be nice to have something written to disk as well.

Sure thing. I’m assuming you’re referring to the examples here. Reported here.

Yes – thanks!

When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__ . Is that a bug?

I do not follow. Can you elaborate please?

Never mind, I didn’t realize that the -c was specific to the example and not for Catalyst scripts in general

If there are more than one channels, is there something to handle this for the -c option?

-c option is for the wavelet_driver, which only produced 1 channel. Are you suggesting we extend that miniapp to produce multiple channels for demostration purposes? Sure, I don’t have any objections there. Shouldn’t be too hard.

Again, thought that -c was a general thing and not specific to this example. Now that I know what’s going on I don’t think that’s necessary to create a multiple channel example at this point.

It would be great to have something available like gridwriter.py and allinputsgridwriter.py

Makes sense. Reported here.

Again, awesome! Will help users immensely IMHO.

What is Frequency for in the Generate Catalyst Script Options window under Global Options?

It is same as every “n-th”. Global options are checked before each extract generator specific trigger options (or live specific trigger options) are checked. Frequency is probably not the right term, but that’s what we’ve used previously so for the lack of a clearer alternative, I left that unchanged.

I guess I’m still not understanding how this global option Frequency will interact with other Frequencies for specific in situ outputs will interact. I haven’t tested yet but if this global option Frequency is 5 and my only writer has a Frequency of 3, say, what time steps will be data be written out?

I guess this adds another question – should the Trigger option be Time or Time Step/Cycle? I know there’s only one option now but hopefully that will change in the future so may as well get it right now .

[utkarsh.ayachit (Utkarsh Ayachit (Kitware))]

@utkarsh.ayachit

I haven’t tested yet but if this global option Frequency is 5 and my only writer has a Frequency of 3, say, what time steps will be data be written out?

In that case, your writer will be activated when timestep_index % 5 == 0 && timestep_index % 3 == 0, thus for the 0th, 15th, 30th, … timestep (or cycle).

I guess this adds another question – should the Trigger option be Time or Time Step/Cycle?

Currently, we only have timestep/cycle based triggers. However, the trigger infrastructure is extensible. We should be able to support other types of triggers in the future, even Python-based ones.

[Andy_Bauer]

I’m just thinking that the Trigger option should be named Time Step instead of Time to be more accurate. In the future I think the first one I’d like to see added is where it’s actually based on simulation time instead of simulation time step. As an example, as close to every half a second in simulation time so if you’re trying to make an animation from images that are dumped in situ that it’s doable, especially for simulations that have variable time step lengths.