PVSM ParaView State - Saving-Loading-Productivity

[patchett2002 (John Patchett)]

Starting with 5.8.1 we have have a plugin that allows a single button click to save a .pvsm state file with a matching .png and empty .txt file all with the same name, different extensions. (saveStateandScreenShot Plugin) I used this capability heavily during a recent project and ended up with nearly 200 of these 3 file sets in a directory that was shared with the team via git. If I wanted to load a state into ParaView I found it most convenient to use my Ubuntu or mac OS file browser, identify the image I wanted to load, then drag and drop the pvsm onto a running ParaView which automatically loaded the pvsm. I’ve done this a lot in the last month and have some possible Improvements to using the pvsm system:

1. When opening in ParaView there is a first order distinction that needs to be made as to whether you are opening data or a state file. Why can’t this be a common open and have paraview (rather than the user) deal with a pvsm vs data file? Wouldn’t that simplify the interface? (I frequently clicked one when meaning the other, navigated for a long time only to find out I picked the wrong thing to open.)

2. When browsing for a pvsm, it would be very convenient if ParaView recognized a same named png and provided an image in the file browser that would allow a user to identify the associated pvsm, maybe even somehow show the contents of the associated text file, maybe in a pane?

3. When loading a pvsm it would be nice if it ParaView didn’t automatically process the state, but rather gave you an opportunity to inspect and maybe alter the pipeline before executing the pipeline (similar to the whole discussion about apply vs. autoapply)

4. When loading a pvsm a window pops up that allows you to adjust the locations of the files that are being read, which is necessary and useful when sharing pvsms among users. Once the files are chosen, ParaView executes the pipeline. When choosing the files it isn’t clear what role each file plays in the pipeline, this makes it more difficult especially when there are many data files to read.

On a PVSM load, I could imagine the pipeline just showing up in the pipeline browser, with the capability of setting the sources (files) in the pipeline browser’s properties panel, before doing an apply rather than the popup. It would be convenient if one could change the reader’s input file from the properties panel forcing the pipeline to be updated with a new data file. (Any source that doesn’t have a proper file path could be made bold and red). I was producing new data files frequently that were on the order of 100GB a piece, it took a long time to load each one, and frequently I would pick the wrong one, after loading a pvsm with the wrong files, my options were to open a new file, get a new pipeline element and change the inputs of one or more filters that would come after, this was particularly difficult for multi input filters, making it easier to just start over. It would have been much easier if I could just change the input filename on the reader.

5. Change the input data file in a reader. This would be generally useful even without the PVSM. I guess there would be good arguments against a data file change in an already applied reader, ParaView is already kind of allowing the results of such a change by allowing the user to pick new data files during a pvsm load, I would guess the pipeline would respond the same when new data files were selected that didn’t have all of the correct scalars/data required for the pipeline.

6. For all filters that take multiple inputs, clearly show somewhere what the inputs are and what their IDs/order are.

[utkarsh.ayachit]

@patchett2002

Why can’t this be a common open and have paraview (rather than the user) deal with a pvsm vs data file?

This is so for one simple reason: when in client-server mode, currently, we only support loading pvsm files from the client-side while load data-files from the server-side. Thus the file-dialog has to navigate different file-systems based on your connection and whether you’re loading state or data.

That brings up several questions:

1. should file dialog support navigating local and remote file-systems seamlessly?
2. should user be able to load pvsm files from local or remote file-systems?
3. should user be able to load data files from local or remote file-systems?

I’d argue #1 and #2 are reasonable, while #3 is beyond ParaView’s scope.

With such a redesign, we can easily support a singular File | Open that opens data files and state files.

@patchett2002

When browsing for a pvsm, it would be very convenient if ParaView recognized a same named png and provided an image in the file browser that would allow a user to identify the associated pvsm, maybe even somehow show the contents of the associated text file, maybe in a pane?

Couple of things to note:
First, opening large PNGs esp on remote sessions is non-trivial, time-wise. It can be slow, esp if the file-system is slow. By your account, you have 100s of state files and associated PNGs – which I am not sure what resolution they are being saved at, but I’d assume HD or UHD. That’d make browsing a slow experience. There’s a reason by Ubuntu generates thumbnail caches for all images in a folder.

Second, if we choose to put the png preview on a side-pane, the workflow would still require the user to click on each of the state files to view the preview – which can get easily very tedious when dealing with 100s of files.

Here’s a counter proposal;

• we extend pvsm to include a thumbnail of fixed resolution 128x128 or 256x256 (typical icon systems on most systems). Having a fixed/small-ish size is key so that it acts like a thumbnail cache.
• the file browser can then render these thumbnails, if present for state files. That’ require some refactoring of the file-dialog, but as allude by my earlier suggestion that is probably overdue.

@patchett2002

When loading a pvsm it would be nice if it ParaView didn’t automatically process the state, but rather gave you an opportunity to inspect and maybe alter the pipeline before executing the pipeline (similar to the whole discussion about apply vs. autoapply)

I am not sure I agree with this one. A state is snapshot of the application. You load a state file to get back to where you left off. You can always change parameters to your heart’s content once the state file is loaded. If you want a customizable state file, you probably want a macro that accepts input parameters from the UI. That’s a different use-case and should probably be tackled differently.

@patchett2002

When loading a pvsm a window pops up that allows you to adjust the locations of the files that are being read, which is necessary and useful when sharing pvsms among users. Once the files are chosen, ParaView executes the pipeline. When choosing the files it isn’t clear what role each file plays in the pipeline, this makes it more difficult especially when there are many data files to read.

Indeed. The dialog is a little ill-organized. Right now it’s categorized by the reader type — which is silly. However, I’d argue it should still be a pop-up dialog, but include the pipeline as follows:

Here, all filenames are listed in a scroll-able view and as the user activates a group, the corresponding reader in the pipeline browser-like widget gets highlighted (and vice-versa). Thus, one can easily view all readers and their chosen filenames as well as their relationships/roles in the pipeline. Note, the user cannot change filter parameters (or other properties on readers), only filenames on readers are shown.

This avoids the need for the application to be in limbo-state after the state file has been loaded to allow the user to make changes.

Change the input data file in a reader.

That is reasonable. It still should be shown on the Properties panel but available as an action in the context menu (same place where we let user change input for filters).

For all filters that take multiple inputs, clearly show somewhere what the inputs are and what their IDs/order are.

This should generally be the case. Let’s report specific issues where things are not clear and we can tackle those.

[theodorebaltis]

This post hit on several common complaints from users at my company when trying to convince them to switch to ParaView from other visualization programs.

So just vocalizing opinion from a larger group as well as giving my personal 2 cents on a few of these.

@patchett2002

Why can’t this be a common open and have paraview (rather than the user) deal with a pvsm vs data file?

This has come up a good bit, new users clicking Open and then being unable to find state files. Personally I don’t think it’s a big deal to keep the actions/dialogs separate, but I would suggest maybe at least renaming Open to Open Data in the toolbar and popup text to better differentiate it from Load State.

But Utkarsh’s response then hit on the biggest complaint from our userbase:

@utkarsh.ayachit

That brings up several questions:

1. should file dialog support navigating local and remote file-systems seamlessly?
2. should user be able to load pvsm files from local or remote file-systems?
3. should user be able to load data files from local or remote file-systems?

Everything we do here is hosted on a server: raw simulation output, post-processing output, cat pictures, etc. I don’t know how common or uncommon that is, but at least for us it’s pretty inconvenient for saving state files. Mapping a few Windows network drives to commonly used directories on the server (or at least one to the server root) is a decent workaround, but you know, people still complain.

As far as I’ve understood, to date this has been a design decision with the idea to keep the state files easily portable. Perhaps this makes sense if the server is only for hosting sim data and is read-only, but otherwise I see saving the state files to the server where the data lives as the most portable solution.

My take on Utkarsh’s questions:

1. Not sure what “seamless” is in your mind, but I would think even an initial popup that basically asks “do you want to see a local or remote filesystem browser next?” (only necessary if connected in client-server) would be satisfactory.
2. Yes, absolutely. Option to save/load pvsm files to/from local or remote filesystems.
3. Not necessary. I could actually benefit from this personally but I acknowledge that it is very likely a fringe and uncommon scenario.

@patchett2002

When loading a pvsm a window pops up that allows you to adjust the locations of the files that are being read, which is necessary and useful when sharing pvsms among users. Once the files are chosen, ParaView executes the pipeline. When choosing the files it isn’t clear what role each file plays in the pipeline, this makes it more difficult especially when there are many data files to read.

@utkarsh.ayachit

Indeed. The dialog is a little ill-organized. Right now it’s categorized by the reader type — which is silly. However, I’d argue it should still be a pop-up dialog, but include the pipeline as follows:

Another issue that’s been commonly brought up. I personally agree that the file-switching mechanism should still be encapsulated in a pop-up dialog, though.

[wascott (Walter Alan Scott)]

@Kenneth_Moreland Thoughts?

[utkarsh.ayachit]

@theodorebaltis

Not sure what “seamless” is in your mind, but I would think even an initial popup that basically asks “do you want to see a local or remote filesystem browser next?” (only necessary if connected in client-server) would be satisfactory.

My thinking is something like this:

The side-pane should make it possible to switch between local and remote filesystems.

[utkarsh.ayachit]

@Kenneth_Moreland had a suggestion offline that we stop supporting saving state files on client and instead only support saving/loading pvsm state files on server side. I think that makes sense. That avoids the need to for the dialog to support browsing local and remote filesystems at the same time and thus minimize confusion.

We would still be saving results from File | Save Screenshot and File | Save Animation on the client-side alone.

[wascott (Walter Alan Scott)]

+1

[mwestphal (Mathieu Westphal (Kitware))]

So something that came up a few time regarding the file dialog was the “time-series” mechanism.
Many users are numbering files without any time series intent, so having an setting to disable it would be nice.

[mconti (Marston)]

Having the ability to load everything from server side while in C/S mode would be a GREATLY appreciated feature. Besides what has been touched on previously, currently images and other data is treated differently than reader data, which doesn’t make sense to me and makes scripting harder than it needs to be.

For example, working in C/S with a windows client and linux server. All data is stored server side.

I create a state file that loads simulation data and then I add in some background images for OSPRay use.

In the state file, the simulation data is saved as network path (linux style), while the images are stored as client path (windows style). Thus, when using this state file to make animations on the server I have to go in to the state file and overwrite the image location path so it can be found.

It’s something I’ve scripted some work arounds for, but seems like a bug to me. Would be great if there was an option to change that behavior. Just my 2 cents.