We need better testing procedures for releases. This would replace our current ambiguous statement to test some of the features in the classroom tutorials. These tests are designed to catch issues that are not found in the nightly testing. Here is my proposal. I figured I would post it here before adding it as a task.
For every major release (5.*.0), minor release (5.13.0) AND every RC1,
1 For every version of ParaView,
Help/ Example Visualizations. Open every example
disk_out_ref. Temp. Volume Render.
Can.exo. Save state. Load state. Save Screenshot (.png). Save Animation (.avi)
Help/ About
2 For one version of ParaView,
Disk_out_ref. Plot over line.
Disk_out_ref. Select a group of cells, extract selection
Disk_out_ref. Find dialog. Find cell 100.
Start trace. Open disk_out_ref. Clip.Create Screenshot. Create Animation. Stop trace. Save macro. Reset Session. Delete screenshot and animation. Run macro. Check screenshots and animations
3 For one version of ParaView, run remote server.
Help/ About. Client and server side.
Disk_out_ref. Memory Inspector.
Disk_out_ref. Change opacity to 0.3.
Can. Save Animation (.avi)
Settings/ Rendering threshold == 0
Disk_out_ref. Change opacity to 0.3.
Can. Save Animation (.avi)
4 For Linux version of ParaView
Pvpython. Create a cone.
PvBatch. Run the trace created above. Run non MPI and MPI.
For every major release and minor release, (No release candidates, as links aren’t in place)
@wascott A lot of these tests seem like they should be automated, image-based tests. What is it they test that we cannot automate? The kind of things we cannot currently automate are things like testing Qt is actually displaying the UI elements that the automations activate by function calls. If that is what you intend to cover (which would be good), we should document what testers should be looking for in particular (missing/disabled buttons, glitchy behavior for small variations, etc.).
Looking through this list, I would say that about half could be (and probably already are) automated. But many probably cannot be easily automated. For example, everything that writes out a screenshot or animation would be tricky to test that the output is as expected. When adding a macro, it’s hard to check that the UI updates as expected. It’s hard to check that clicking on help brings up the expected help.
Although a lot of these tasks are simple and already automated, I think the point is to verify that something doesn’t go wonky with the application. I think @wascott’s intention is to run through some basic operation with a human watching as a final sanity check before release.
Ken nailed it. The point is to basically install every download for release (the file, not just a nightly build), check Help menu links, check that it works remote server, make sure volume rendering and opacity works, make sure the trace recorder works correctly, make sure save and load state works, make sure Python works (it silently “hides” remote server unless you run Help/About, or run something pithony). If these can be automated, great. That’s an implementation detail. But they should still be tested.
Did we miss specific issue without these tests in the past ?
In any case, what can be automated should be automated, and certain test should be put in the superbuild in order to test the actual released packages.
Manual test should focus on what cannot be tested automatically.
Regarding the list above, I see almost only thing that can be tested automatically, either in CI or in the superbuild CI. Some are probably already tested, but I did not check.
Example Visualizations: Superbuild CI
VolumeRender: CI
Save/Load: CI
Help/About: CI
PlotOverLine: CI
Selection: CI
FindData: Worth testing in superbuild CI
Trace: Worth testing in superbuild CI
Client/Server: CI
MemoryInspector: CI
Remote/Local rendering with opacity: CI
pvpython/pvbatch: superbuild CI
Help: superbuild CI
However, what is missing IMO is the testing of the integration of the release in the desktop, especially on Windows and MacOS which comes as installers. We can see in the “release” issue that this is tested though:
As we can see, there is even some specific steps to check some of the scenarios you are highlighting.
So unless I’m missing something, the way forward is to patch testing holes that may be present in the CI or superbuild CI, but not to add more manual testing.
@wascott I am a little confused here with the mention of “Disk_out_ref” and “Can”. Is there an implied reset session after each of these steps and a loading of these datasets?
Sorry @cory.quammen, I think that was a cut and paste error. Maybe something like this?
What I think I was trying to show is that remote server works properly for things that sometimes break remote server. First is making sure Python is working (this is the best test I know for a bad server side Python). Make sure opacity works properly, then that animations work when rendering is occurring locally and remote server.
Help/ About. Client and server side.
Disk_out_ref. Memory Inspector.
Disk_out_ref. Change opacity to 0.3.
Reset Session (or just start another run, which is fine.)
Sure, we can do that. I think g1s1 is too big to add as an example data file in the binary, but we can add it to the test data so that it is widely available.
Is g1s1 sensitive or releasable? I thought there was something about that data that made it not appropriate for open release, but maybe something changed (or I’m thinking of a different dataset).