Buildslave feature idea : Test regexp

Dashboards
1

It would be nice to have a feature to select which run to test instead of running all test.
Something to pass to ctest -R would be enough.

I know that one can edit the CMakeLists.txt to disable all the test except a few, but it is not practical.

2

This has been discussed on our side a bit in the past, but obviously never implemented on the buildbots. @utkarsh.ayachit @ben.boeckel is this potentially feasible with gitlab CI?

3

should be doable in gitlab-ci via variables

4

You can add a WIP commit which modifies the ctest_test.cmake script to pass extra arguments to ctest_test() to limit your testing.

How else could we pass variables to an MR build that doesn’t require a diff to the MR (that needs ripped off at some point which will rerun full CI anyways)?

5

Adding a WIP commit is doable but not practical. tbh, I end up running all the tests and stalling other people work just to avoid pushing a WIP commit.

How else could we pass variables to an MR build that doesn’t require a diff to the MR (that needs ripped off at some point which will rerun full CI anyways)?

Can’t arguments be passed from buildslave to the ctest call ?

6

@ben.boeckel : I couldn’t find this file, could you give some details on how to do this ?

7

With gitlab-ci, our pattern is to use a CTest script that you can edit in your MR to restrict the test suite (please do it with a WIP commit so it doesn’t affect master).

8

I’m still confused on how one is supposed to do that.
Could you open a mr just to show this ?

9

You can edit the .gitlab/ci/ctest_test.cmake to have:

ctest_test(
  PARALLEL_LEVEL "${nproc}"
  RETURN_VALUE test_result
  INCLUDE "SomeTestRegEx" # WIP: For faster turnaround.
  EXCLUDE "${test_exclusions}")
1 Like
10

I’ve been trying without success here : https://gitlab.kitware.com/paraview/paraview/-/merge_requests/4273/diffs#note_802016

11

After a discussion with @ben.boeckel on the linked MR, this is currently not possible in any practical way.

That would be a very nice to have, especially for WIP MR that just want to check their tests are passing before running the whole test suite.

1 Like
12

With gitlab-ci being adopted now, it would be great to add this as well :slight_smile:

13

Another related feature request: it would be nice to be able to run the buildbots without running tests, only to see if there are build errors/warnings.

14

I’m not really keen on adding features to buildbot. Hopefully it isn’t used (much) in a few months at all.

You can add INCLUDE_REGEX to ctest_test.cmake in a WIP commit to your MR to restrict the set of tests. But it really should run all of them eventually to make sure you’re not breaking things.

15

I’m not really keen on adding features to buildbot. Hopefully it isn’t used (much) in a few months at all.

@nicolas.vuaille meant to be able to only build and not run the tests with the ParaView ci, being buildbot or gitlab.

You can add INCLUDE_REGEX to ctest_test.cmake in a WIP commit to your MR to restrict the set of tests. But it really should run all of them eventually to make sure you’re not breaking things.

Great, I will try it soon.

16

clang-tidy will essentially do that once it’s on gitlab-ci. It will be Linux only (we don’t have enough macOS and Windows capacity to have a dedicated build-only and running the tests is, by far, the more common case, so that will remain the default).