Folks, I wonder if we should restructure our buildbot/dashboard code to not look at site specific build options and test exclusions in the buildbot repo, but in the source repo itself?
The reasons for this are as follows:
Let’s say I am adding a new module to VTK and this is an optional module. Now, to enable it on certain dashboards I have to separately update the buildbot repo and then test my MR. While this is minor annoyance, it can be a major issue if this new module is already in VTK and turning it on without merging my MR has dire consequences (@aron.helser can attest to this). Won’t it be nicer if I could turn on/off the module in by MR itself?
With “release” branch now supported, we end up with cases of different build options between releases and master. Short of setting up new builders for each of the branches, which has it’s own issues, currently we have to manually toggle the options when building release binaries (just ask @cory.quammen). With the suggested change, the config will be on the branch itself.
Test exclusions are added to builbot repo and not the project. Updating the exclusions can be slow (just ask @martink), can got out of date (eg. test names change but exclusions linger) etc.
It’s hard to determine exactly what cmake options are turned on/off on a particular dashboard. The suggested change will make it so that these are all simply in a directory in the source itself (except paths to external libraries etc, which will still be in the buildbot configs).
I’ve had some time to think about how to do this now. Here is roughly what I am thinking:
Implementation-wise it is fairly straightforward. We include JSON files for each build in the repository at $somepath. We add 3 build steps, one to upload $somepath/$currentbuild.json to the buildbot master, one to read it and add the cache settings, exclusions, etc and one to delete the file on the server after we are done with it.
We need a json file rather than a cache because it will contain at minimum cache settings and test exclusions.
The question then becomes how much do we move to the source tree. All cache settings? We would lose features if we did this and our builds would be named something like eeloo-build1 and eeloo-build2. What about machine-specific paths to libraries? Do we want those in the repo?
I feel CMake config files (the ones that can be passed in to cmake -C) would be preferable to json for the following reasons:
they do not introduce any new concept in the project source repos. All developers of the project need not be aware of what the buildbot configuration files are.
they are useful for a broader developer/user community. One can simply run cmake -C $somepath locally to build with flags similar to any dashboard.
The test exclusions are indeed the issue. Maybe there’s a way to handle that without requiring json.
The project repo can have feature specific config files too. e.g. $configs/mpi.cmake, $configs/ospray.cmake, etc. Since one can pass multiple -C arguments to cmake, based on the features enabled on the builder, appropriate cmake config file could get imported.