Extracts/Exports and Catalyst Python Scripts

[utkarsh.ayachit]

First pass of this has now been merged in master.

• User guide updates
• Catalyst Python Scripting details

[wascott (Walter Alan Scott)]

Finally got around to reading this. Feels like reading a Michener book. +1. Well done.

[utkarsh.ayachit]

sorry! this thread got long very quickly!

I think what we need next is a set of short tutorials to get folks started with these extract generators and catalyst python script changes.

[Andy_Bauer]

I like these changes a lot! This looks like a complete redo that’s probably been overdue for some time. Some ways to possibly improve:

1. Would it be possible to add in a hover to the Extracts Generators->Data->* ? I know what many of them are but not sure what things like VTPD is. I see that it’s greyed out for my specific output so maybe it doesn’t matter though.
2. Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.
3. For the example it would be nice to have something written to disk as well. I added a dataset writer manually and probably didn’t do it the proper new way but others may have trouble doing that.
4. When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__. Is that a bug?
5. If there are more than one channels, is there something to handle this for the -c option?
6. It would be great to have something available like gridwriter.py and allinputsgridwriter.py that could write out the full grid if the default input channel is used and for any number of channels are used, respectively. I think a number of people have used these in the past to help them get going in the Catalyst workflow.
7. What is Frequency for in the Generate Catalyst Script Options window under Global Options?

I very briefly skimmed the user guide updates so apologies if I missed something in there. The example did work nicely for me.

[utkarsh.ayachit]

Firstly, thanks for trying it out, Andy! Very much appreciated!

Would it be possible to add in a hover to the Extracts Generators->Data->* ?

Indeed. Reported here

Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.

Good point. I wonder if we should really not burden the user with which flavor of writer to use, instead have single entry for VTK-XML (or something more readable) and then create the *.vt??? variant based on the input dataset type. Thoughts?

For the example it would be nice to have something written to disk as well.

Sure thing. I’m assuming you’re referring to the examples here. Reported here.

When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__ . Is that a bug?

I do not follow. Can you elaborate please?

If there are more than one channels, is there something to handle this for the -c option?

-c option is for the wavelet_driver, which only produced 1 channel. Are you suggesting we extend that miniapp to produce multiple channels for demostration purposes? Sure, I don’t have any objections there. Shouldn’t be too hard.

It would be great to have something available like gridwriter.py and allinputsgridwriter.py

Makes sense. Reported here.

What is Frequency for in the Generate Catalyst Script Options window under Global Options?

It is same as every “n-th”. Global options are checked before each extract generator specific trigger options (or live specific trigger options) are checked. Frequency is probably not the right term, but that’s what we’ve used previously so for the lack of a clearer alternative, I left that unchanged.

[Andy_Bauer]

Thanks!

@utkarsh.ayachit

Firstly, thanks for trying it out, Andy! Very much appreciated!

Would it be possible to add in a hover to the Extracts Generators->Data->* ?

Indeed. Reported here

Also, for the Extracts Generators->Data->* it seems like it’s a bit inconsistent to me that, for example, it lists VTI, but is actually a pvti output.

Good point. I wonder if we should really not burden the user with which flavor of writer to use, instead have single entry for VTK-XML (or something more readable) and then create the *.vt??? variant based on the input dataset type. Thoughts?

Not a bad idea to just say Parallel VTK XML output and create the .pvt? variant, if it can be done in a non-confusing way. We’ve been doing it for so long though with .pvti, .pvtu, .vtm, etc. though that maybe we’ve trained the users too much in that way of thinking. I’ll leave it up to you though.

For the example it would be nice to have something written to disk as well.

Sure thing. I’m assuming you’re referring to the examples here. Reported here.

Yes – thanks!

When using the -c to change the registrationName for the channel, it keeps that name throughout and when writing out a file to disk it uses that name for __CatalystChannel__ . Is that a bug?

I do not follow. Can you elaborate please?

Never mind, I didn’t realize that the -c was specific to the example and not for Catalyst scripts in general

If there are more than one channels, is there something to handle this for the -c option?

-c option is for the wavelet_driver, which only produced 1 channel. Are you suggesting we extend that miniapp to produce multiple channels for demostration purposes? Sure, I don’t have any objections there. Shouldn’t be too hard.

Again, thought that -c was a general thing and not specific to this example. Now that I know what’s going on I don’t think that’s necessary to create a multiple channel example at this point.

It would be great to have something available like gridwriter.py and allinputsgridwriter.py

Makes sense. Reported here.

Again, awesome! Will help users immensely IMHO.

What is Frequency for in the Generate Catalyst Script Options window under Global Options?

It is same as every “n-th”. Global options are checked before each extract generator specific trigger options (or live specific trigger options) are checked. Frequency is probably not the right term, but that’s what we’ve used previously so for the lack of a clearer alternative, I left that unchanged.

I guess I’m still not understanding how this global option Frequency will interact with other Frequencies for specific in situ outputs will interact. I haven’t tested yet but if this global option Frequency is 5 and my only writer has a Frequency of 3, say, what time steps will be data be written out?

I guess this adds another question – should the Trigger option be Time or Time Step/Cycle? I know there’s only one option now but hopefully that will change in the future so may as well get it right now .

[utkarsh.ayachit]

@utkarsh.ayachit

I haven’t tested yet but if this global option Frequency is 5 and my only writer has a Frequency of 3, say, what time steps will be data be written out?

In that case, your writer will be activated when timestep_index % 5 == 0 && timestep_index % 3 == 0, thus for the 0th, 15th, 30th, … timestep (or cycle).

I guess this adds another question – should the Trigger option be Time or Time Step/Cycle?

Currently, we only have timestep/cycle based triggers. However, the trigger infrastructure is extensible. We should be able to support other types of triggers in the future, even Python-based ones.

[Andy_Bauer]

I’m just thinking that the Trigger option should be named Time Step instead of Time to be more accurate. In the future I think the first one I’d like to see added is where it’s actually based on simulation time instead of simulation time step. As an example, as close to every half a second in simulation time so if you’re trying to make an animation from images that are dumped in situ that it’s doable, especially for simulations that have variable time step lengths.

[Andy_Bauer]

I’ve been playing around with this a bit more and there’s no way that I can see to output the old style Catalyst Python scripts. Is that correct or am I missing something? It seems like using Catalyst 5.9 will not be backwards compatible.

The other thing is that the screenshot output for Catalyst doesn’t quite fit in the pipeline browser thinking that we’ve presented to the user in the past, at least the way it’s presented right now:

I wonder if instead it should be a connection on the right side with all of the visible filter outputs being connected to it. Something like the following:

[utkarsh.ayachit]

@Andy_Bauer

I’ve been playing around with this a bit more and there’s no way that I can see to output the old style Catalyst Python scripts.

That is correct. We no longer support outputting old style scripts.

@Andy_Bauer

It seems like using Catalyst 5.9 will not be backwards compatible.

That is not correct. ParaView 5.9 still supports loading old style co-processing scripts. There are tests prefixed with Catalyst::Legacy:: that test loading old Catalyst scripts in the new version.

ParaView has never supported generating state files or scripts that can be loaded in older versions of ParaView and that indeed is the case with 5.9.

@Andy_Bauer

The other thing is that the screenshot output for Catalyst doesn’t quite fit in the pipeline browser

See this issue. One proposal is to add an icon where the eyeball is next to image extract generator to give an indication of which view is being saved out by that image extract generator.

Note the image extract generators are not really connected to filter outputs, but more directly to the view. Thus providing a clear indication of which Image Extract Generator is associated with which view, or at least associated with the active view, should be a good place to start.

[Andy_Bauer]

My point that Catalyst 5.9 will not be backwards compatible is that if we link a simulation code that used to work with Catalyst 5.8, without changes, to Catalyst 5.9 there is a decent chance that things will not work. The reason I say that is if we have a Catalyst Python script that script may need to be updated from the 5.8 Python API to the 5.9 Python API (if it only has data extracts probably not but if it has screenshot/image output it probably will). That new script that we generate in ParaView 5.9 won’t work with our existing vtkCPPythonScriptPipeline class.

As for the the screenshot output, yeah, I meant to indicate that it’s not really the filter output but the open eyeball of the filter output. And still, this is just for the active view. Basically I’m just trying to make the point that we’re really trying hard to provide a new type of gui functionality in the pipeline browser that we haven’t in the past. That’s not my skill set on how to do that properly but wanted to point it out so that people much, much better than me at HCI can look into a good way of doing it.

[utkarsh.ayachit]

@Andy_Bauer

The reason I say that is if we have a Catalyst Python script that script may need to be updated from the 5.8 Python API to the 5.9 Python API (if it only has data extracts probably not but if it has screenshot/image output it probably will).

Yes, if you want an adaptor to use the new style Python scripts exported in ParaView 5.9, you will need to update your adaptor code.

[Andy_Bauer]

Another question on this – is there the Cinema functionality to iterate over Slice offsets and/or Contour iso-surface values? I can’t seem to find how to set that in the GUI. I can see the camera model and moving that around still.

[terryturton]

This would be very useful functionality, especially in Catalyst. Working with ECP applications, this is a common in situ request – a slice or contour every n timesteps.

[utkarsh.ayachit]

@Andy_Bauer

Another question on this – is there the Cinema functionality to iterate over Slice offsets and/or Contour iso-surface values?

This has been removed for this version. ParaView 5.9 sacrifices some Cinema capabilities with the goal to make it easier, usable and consistent in various modes of operations.

This version does not support composable images since Cinema project is working on coming up with a new spec for the Cinema composable image set (CIS). Once CIS file format is finalized and a writer API is available from the Cinema Science project, ParaView will use that to add a new Image Extract Generator of type CIS and support “layers” allowing users to toggle various props in the view. It’s unclear if we’ll support various slice offsets, or iso-values as it’s too custom a use-case. We might just rely on users to create multiple contour or slice filters for same effect – but that can be figured out when we get closer to implementing it.

More details here.