With the aim of improving and streamlining communication about bugs and problems with ParaView, we are considering adding issue templates for regular use to ParaView’s GitLab project.
An issue template contains boilerplate text with sections that specify what kind of information to fill in that will be helpful to the person reviewing the issue. An example for a bug report might be:
Detailed description with screenshots as appropriate
GitLab lets you have a number of different templates for different purposes. For example, you could have one for bug reports, one for feature requests, and one for usability questions that could redirect people to Discourse.
GitLab also let’s us set a default issue template. The default could be some text that simply strongly suggests people select one of the existing templates.
I think adding one or more default templates would be very helpful to direct reporters to provide the necessary information to identify and fix issues.
If possible, I think it would simplify things if there was just one (default) template that covered the standard issues most users would encounter. The list of sections you’ve presented above seem like a good set to use. I suspect you wrote those with bug reporting in mind, but I think they will work well for feature requests as well (or at least well enough to avoid the complications of a second template).
A suggestion: you can use html-style comments to add instruction text to the template that won’t show up in the description. You could use that to instruct reporters how to fill out the template or redirect questions on use to discourse.
<!—- Note: text in HTML comments like this will not be displayed in the issue description. —>
<!— This form is for reporting bugs or feature requests for ParaView. For questions on how —>
<!— to operate ParaView, please post questions to https://discourse.paraview.org/ —>
<!— To report a bug or feature request, please fill out all of the sections below. —>
Another note in case you have not played with it before, GitLab supports quick actions that can automatically set up the state of the issue when submitted. You could, for example, automatically add a label to put the newly created issue into triage, e.g., /label ~”triage:awaiting-confirmation”
Spectacular idea. Totally on board. Your list is basically the list I use to train all new ParaView support folks here.
One thing to note is back a few years I found that the search function only works searching in the body, not the title. Thus, if the title says Volume Rendering is broken on a Mac, but you don’t say that in the body, you can’t search for it. Either we need to verify that I am correct, and/or add a comment to this affect in the template.
Would others agree? If I’d been required to fill out a bunch of fields, I probably wouldn’t have submitted any of these issues. As long as it’s clear that the template is a suggestion to aid in producing a good issue, that’s fine, but if it becomes a gate/hurdle then it will dramatically impact issue reporting.
Well, all a template really does is pre-populate the issue’s description text, so the submitter is free to ignore or delete whatever they want. And it is fair to suggest adding comments to when a section can be safely ignored (e.g., you don’t need to post a dataset if the steps do not require one). The real question is how strict ParaView developers will be when deciding what issues to address. Given my experience with ParaView developers, I doubt they would dismiss an issue outright just because it didn’t follow the template exactly. This is just a tool to communicate.
To play, uh, the Devil’s Devil’s Advocate (Angel’s Advocate?), the template would make the example issues above more helpful. For example, in the issue of the colorbar title overlap, I tried replicating the issue on my system and got this:
If I was the developer addressing this issue, I might just close the issue thinking it was fixed or perhaps add a few pixel spacing. Neither response would fix the issue. If the issue followed the template, there would be instructions to guide me to the same setup as you (or maybe a pvsm file to automatically set up the state). There would also be information about your system setup (ParaView version, OS, etc.) that might affect how things get rendered and explain why I might see something different.
Each submitted issue is requesting a significant amount of time on a developer’s part. I don’t think it is unreasonable to request a submitter to spend an extra 10 minutes describing an issue to save a developer 30 minutes trying to understand and replicate it.
If that’s all a GitLab template is, then that’s good – at a former employer we used a system where there were fields that if not filled out you couldn’t even submit an issue. And, as would quite often be the case, one wouldn’t have time right now to fill in all the details - or wouldn’t be able to share the data (e.g., classified data that could take months to get a declassified version approved for sharing), and thus would just never submit the issue.
(A particular case I remember (an internal tool, not ParaView) was one where there was a broken “help” link, but I couldn’t submit the issue without providing “datafiles” and a screen-recording of the issue when all one needed to replicate was click File->Help->Open Manual - so I didn’t)
Not knowing exactly what a GitLab template entailed I just wanted to make sure this was considered – since it sounds like templates don’t work in this obtrusive way it’s probably a good idea (i.e., it works in the “just a suggestion” way).
You’ll note that I’ve followed the template in the first post “to the ‘T’”. One comment I’ll have is that it feels like I could have simply attached a screenshot of the issue, a ParaView state file, and the copy-paste of the Help -> About... -> Copy to Clipboard, rather than a 19-step “steps to replicate”. Instead you’ll note that I saved the Sphere source to a PVD file and attached it, since the template says Dataset that shows the problem. Again, just playing Devil’s Advocate to help identify what would be helpful in the template.
My thoughts on the template:
ParaView version and operating system should maybe request the copy/paste from the About dialog that I pasted instead (would it have been better for me to just type ParaView 5.12.0, Windows 11?)
Assuming that #1 is done, do we still need Configuration section?
Feels odd to have Brief Description, then information about version / configuration, then Observed Behavior and Expected Behavior… why the separation, also, feels redundant – especially when there’s then a Detailed description section… why do I need to describe the issue in four different, separated (by distance) sections? Maybe just a single heading for Brief description of observed behavior and expected behavior and then another heading for the Detailed description?
Regarding Steps to reproduce and Dataset… what is the optimal case? There’s nothing that’s pointed me to providing a state-file… I assume those aren’t helpful then? If helpful, feels like this should be called out.
I like the suggestion to copy/paste the about dialog information. That is easy to provide and could provide clues the submitter may not have considered. Two caveats about that though:
The operating system is missing from these details. It’s probably tricky to report that reliably in the about box information. The submitter will still have to provide that.
There are cases where the about box is not available. For example, if the problem happens in batch mode or when running Catalyst. Maybe the template can have a hidden comment to do this copy/paste where possible.
I agree this feels redundant. Perhaps these two items should be combined.
Specifically describing the observed and expected behavior separately from other information is a common feature of bug reporting advice (see for example here, here, or here). Although it can be a bit pedantic, it can be a helpful tool for communicating what the submitter needs and expects.
It would be nice if comments in the template directed submitters to think ore about the “expected behavior” other than just the negation of the observed behavior. For example, if the observed behavior is “ParaView crashes”, a useful expected behavior may say something like “The reader should detect when the file has field arrays of the wrong size, issue a warning about the inappropriately sized arrays, and continue producing the data set sans the field in question.”
Back in your example, surely there is more to say than the components never overlap. How do you want it to look? How should it behave as you turn on/off the above color range. You could say something like this: “The horizontal title on a vertical colorbar should be placed such that the baseline is placed just high enough for the descenders to clear the colors of the bar. This should be true regardless of whether the above range color is shown. This means that when Use Above Range Color is turned on, the label must be moved up, the color bar must be moved down, or the color bar must be squished to make room for the extra box of color.”
To avoid repeating information, we could rename this section to Additional Information.
I personally think listing the steps to reproduce is better for several reasons.
Seeing a list of steps helps the developer understand what the submitter is trying to do.
Following the steps sometimes gives the developer clues on where things go wrong.
When the bug causes ParaView to crash or other sideways behavior, it can be more difficult to diagnose when all you have an impossible to read state file that just makes things crash.
Some bugs happen from misconfigurations from the GUI that can get coded into the state file. For example, a 3D widget might be sized wrong if the client application misjudges the bounds, and the size could get written to the state file. In this case, the state file will still load wrong even if the bug gets fixed.
Asking for a list of steps encourages submitters to minimize the problem to the think that is causing the problem, which again helps the developer find where the bug is.
Although ParaView has gotten much better about passing state files between systems and versions, it can be that the state file itself stops working.
All that said, state files are not unhelpful. Maybe there is some wording to encourage listing the steps but alternately supplying a state file if there is an issue with that.
A few disjointed thoughts.
*If we mention the About help dialog, be sure to ask for local and remote section.
About doesn’t exist for pvbatch and pvpython.
At the risk of getting yelled at, I think we are getting into the weeds. The goal isn’t to argue if replication steps or a state file are better. The goal is to provide the developer with at least the following: ParaView Version, OS, what program, if ParaView is builtin or remote (and how many ranks), and steps to replicate.
Having reminders and suggestions to provide this is a good idea. It won’t be a requirement, just a suggestion.
There are lots of ways to provide a good bug report. Lets not get tangled in the weeds. Only stating “My dataset crashed” or “My image doesn’t look right” or “Here is a discussion elsewhere” really isn’t one of them.
Alan
My only experience with issue templates are GitHub templates ala MOOSE, which has three required fields and a fourth, optional section. It feels about right and they generally get pretty good bug reports.
I guess my thoughts are just to have a template that’s a bit more concise, yet clear as to what would be beneficial to provide.
Title
Description
A clear and concise description of the problem with screenshots of the issue, if applicable. (Note: A missing feature is not a bug).
Steps to Reproduce
Steps to reproduce the behavior (include example datafiles and/or state files that demonstrate the behavior)
ParaView configuration
If using the GUI client… (link to instructions)
If using pvbatch… (link to instructions)
If using Catalyst… (link to instructions)
I guess my thought is that, if there are just a few fields (three not including the title, in this example) then you can probably make them required fields as MOOSE has. But eight fields just feels like a lot to ask for, even if not required.
