Version test documentation

Development
1

With in situ becoming more prevalent I was wondering about the possibility of documenting the test status of ParaView releases. The reason for this is as we push for better testing in our simulation codes, especially production codes, that means we need to get better at demonstrating how well these simulation codes are tested (e.g. code coverage, unit tests, etc.) but that may also mean how well third part libraries are tested. For something like Catalyst it would be useful to point to some document saying that for release vX.Y it passed some amount of testing to help build confidence in the overall simulation+in situ package.

The reason that I believe that this is a bit different than just using ParaView as a post-processing tool is that simulation codes that use Catalyst may choose to stick with a specific version of Catalyst and thus comparing between results between Catalyst versions is not a simple task. Even with the Catalyst V2 API where users could swap out versions of Catalyst, I’m not sure how willing users would be to rerun an entire simulation to do this type of checking.

2

Are you referring to testing of Catalyst with specific codes or just Catalyst/ParaView itself?

3

Just Catalyst/ParaView itself. Kind of a snapshot of the dashboards (with maybe VTK included) at the time of a release. It’s all kind of vague what I’m thinking here but you can imagine that if you’re running a simulation code that’s used to design nuclear reactors, planes, or something else with rather critical performance needs that documentation along these lines might be appreciated.

4

Though I think this is a well-intentioned idea, dashboard data doesn’t tell the whole story. For example, if all tests pasts, there’s a non-zero chance that a particular integration of Catalyst may turn up new bugs. In fact, we already know there are bugs that don’t yet have corresponding failing tests - they’re in the bug tracker.