Reducing wait time of testing request

Dashboards
1

Particularly for ParaView, job queue is very long on some buildbots (10+ hours) and seems related more to testing than configuring/building.
It can be frustrating to be able to run tests only once a day when deadlines are very short.

Here are some ideas:

  • upgrading hardware of the slowest buildbots
  • add an option to run only some tests based on regex (generally sufficient for WIP MR)
  • run only tests impacted by the modifications (seems difficult to implement though)
  • add an option to set the priority of the request

What do you think?

2

A few other ideas :

  • Rework test that are more than 20s long to improve timings
  • Add an option to run only pv, pvcs or pvcrs tests
  • Improve connection time for pvcs and pvcrs tests
  • Reduce default timeout of 180s to 60s
  • Add an option for only building, not testing
3

Folks can also get a little smart about testing.

  • if fixing doc, no need to run tests
  • if fixing a warning or smaller fixes, just use -i to limit to a builder or two
  • try to update to latest master before requesting tests or do Do: test --merged to avoid running tests on too old a branch
  • when adding a new test, try to see if it’s easier to combine the test in an existing test rather than adding a whole new one.

we do need to update/replace nemesis. it has too many builders right now.

4

do Do: test --merged to avoid running tests on too old a branch

Should this be default ?

5

dres had an issue with a macOS dialog appearing after ParaView crashed that prevented additional instances from running, hence tests were essentially blocked. I prevented these dialogs from appearing (hopefully) by applying the command

defaults write org.paraview.ParaView NSQuitAlwaysKeepsWindows -bool false

dres seems to be chugging along faster now as a result.

6

I am not sure. I’d much rather people rebased their topics to latest master before running tests manually. I prefer things to be explicit, but that’s just my preference.

1 Like
7

nemesis as an 8hour delay currently.
It can’t handle two build of ParaView imo.

8

upgrading hardware of the slowest buildbots

The replacement for nemesis is sitting in an office here waiting to be made available (another project is using it as a Windows box). I’ll probably get around to swapping them when Windows 10 makes its sweep across the machines.

add an option to run only some tests based on regex (generally sufficient for WIP MR)

A WIP commit to comment out unrelated tests can help here. Just remember to remove it before merging (adding WIP as a prefix to the commit allows the robot to enforce this remembering for you).

run only tests impacted by the modifications (seems difficult to implement though)

Indeed. I’ve thought about this, but haven’t come up with a good way of doing it. In ParaView’s case, all the XML tests end up getting affected since paraview the executable ends up depending on “everything”.

add an option to set the priority of the request

Without a central priority queue, this is just going to be abused.

Note that I’d really like to migrate to GitLab-CI at some point and allow more hardware to share a given build configuration, but that is probably at least a year down the line at this point for VTK/ParaView.

9

Great !

That’s actually quite smart ! I should have thought of that.
It is not that easy to do though, as you need to modify different part of multiple CMakeLists.txt, unless I’m missing something.

10

How easy it is depends on how long you’re willing to wait for results for your test(s). Though that is also a function of others doing the same to reduce their test load…

11

One thing I’m doing right now that may be helpful to faster overall dashboard turnaround time is to run an initial round of tests on a faster machine, such as taanab, and fixing any warnings/errors it turns up before running the tests on all dashboard machines.

12

Note that taanab is mostly faster because its build setups are just much smaller than any other machine’s builds.

13

I have an MR up to increase the branch stable timer for master and release branch so that the master builds are not prioritized over MRs too quickly. that should help with turn around time during work hours

14

That’s fine - it’s still useful initial feedback for building errors and warnings before committing the slower builders to a chunk of work.

15

What would be great is if master and release branches could be killed if there is an MR waiting. I have no idea how to implement that or if it is possible.

16

There are remnants of me trying to get Do: test --stop to stop an MR build that has already started, but that never got anywhere. The “Stop build” button on the web interface is “in there” and not easily triggerable from other code.