With in situ becoming more prevalent I was wondering about the possibility of documenting the test status of ParaView releases. The reason for this is as we push for better testing in our simulation codes, especially production codes, that means we need to get better at demonstrating how well these simulation codes are tested (e.g. code coverage, unit tests, etc.) but that may also mean how well third part libraries are tested. For something like Catalyst it would be useful to point to some document saying that for release vX.Y it passed some amount of testing to help build confidence in the overall simulation+in situ package.
The reason that I believe that this is a bit different than just using ParaView as a post-processing tool is that simulation codes that use Catalyst may choose to stick with a specific version of Catalyst and thus comparing between results between Catalyst versions is not a simple task. Even with the Catalyst V2 API where users could swap out versions of Catalyst, I’m not sure how willing users would be to rerun an entire simulation to do this type of checking.