Superbuild: one project working, similar project not

[cobo (Cornelis Bockemühl)]

One of my projects (derived from Paraview and including some additional modules and plugins) seems to work now also with the “superbuild” for generating a Windows installation package.

From what I understand so far, there are different steps happening:

• first a “build” run for the main project and all the depending projects, from boost to Paraview to whatever. The result of this run ends up basically in the sb_target/superbuild directory and it’s subdirectories.

• then an “install” run - not really an installation, but the “copy with extras” that is called “installation” in the cmake language. This step is basically supposed to move those files that will finally go to the package into the sb_target/superbuild directory tree.

These two steps are done during the “cmake” and the “ninja” runs.

The final step, started with ctest -R …, brings finally the files for the package into still another directory tree, rooted at sb_target/_CPack_Packages.

This works nicely for that first project now, but for a very similar project that is based on the first one the second step fails: the “install” brings the files coming from that project not to sb_target/install, but to sb_target/superbuild/second_project/build/install. So the third step does not find them in the right place!

And the question is WHY?

First of all, in the CMakeLists.txt of the projects themselves (not the superbuild projects), I have in both cases this line:

set(CMAKE_INSTALL_PREFIX “${CMAKE_BINARY_DIR}/install”)

(ok, in the first project, that line still has the following - no real idea what the difference is:
set(CMAKE_INSTALL_PREFIX “${CMAKE_BINARY_DIR}/install” CACHE PATH “”
could THIS be the crucial difference??)

In both superbuild target directories I find the following entry in the CMakeCache.txt:

CMAKE_INSTALL_PREFIX:PATH="<sb_target>/install"

with <sb_target> explicitly being the correct path where the sb “install” step should put it’s stuff.

And in the same superbuild target directories I also find a cmake_install.cmake file - and there things are different:

first project - where everything works fine:

if(NOT DEFINED CMAKE_INSTALL_PREFIX)
set(CMAKE_INSTALL_PREFIX “<sb_target>/install”

while in the second project - where the “install” goes to the wrong location it is:

if(NOT DEFINED CMAKE_INSTALL_PREFIX)
set(CMAKE_INSTALL_PREFIX )

So obviously it comes from this cmake_install.cmake script - but how does it get in there? Because from all that I am reading here and there, my only conclusion is that “somehow” the setting from the project would be written there - but that is the same in both projects!

Also I have no explicit “install” commands in any of my own CMakeLists.txt, so only what happens inside the different paraview and vtk module and plugin generation functions is happening - which all do not interfere with the CMAKE_INSTALL_PREFIX as far as I have realized so far.

So I am lost again - and this after so many weeks of finally trying to understand what this miraculous “cmake installation” actually does! Well, I learned that it basically writes the cmake_install.cmake script - which is already good to know. And with this knowledge I could also see that the wrong “install” target comes somehow from that file. Which in turn means for me that the problem does not lie in the superbuild, but in the initial project already. Only that I do not see a difference there in what I am doing - so why the difference in the result?

[cobo (Cornelis Bockemühl)]

Looks like I found a solution myself - even if I do not understand it!

In the above explanations, I pointed out that there is only one little difference between lines in the first (working) and the second (not working) project:

1. set(CMAKE_INSTALL_PREFIX “${CMAKE_BINARY_DIR}/install” CACHE PATH “”)
2. set(CMAKE_INSTALL_PREFIX “${CMAKE_BINARY_DIR}/install”)

And I said that I cannot imagine how that should make the difference - and I still cannot really believe it!

Still I thought that I should remove also this last little difference - and - miracle!! - now the “install” step copied the files to the correct location, so the packaging could find them there, generate an installation package that could even be installed to end with a working software.

So the problem comes down to my not-understanding of the obvious difference between “variables” and “cache variables”. The only thing that I understood so far is that you can override cache variables with command line parameters, but I did not use any such thing myself. Ok, it might happen somewhere deep inside the superbuild. But then the fact that in the CMakeCache.txt the variable was correct in both cases, while only the cmake_install.cmake file had the appearing difference somehow told me: “it cannot be a cache problem”. But then - it looks like it was!

Of course there is now no difference in the CMakeCache.txt files, but there is indeed one in the cmake_install.cmake.

Makes no sense to me, but maybe the world has to keep certain secrets from being understood by my curiosity - like there are still so many everywhere!

Or just trying the “cmake philosophy”: “Dont ask any more questions once your script is working!”

[ben.boeckel (Ben Boeckel (Kitware))]

@cobo

(ok, in the first project, that line still has the following - no real idea what the difference is:
set(CMAKE_INSTALL_PREFIX “${CMAKE_BINARY_DIR}/install” CACHE PATH “”
could THIS be the crucial difference??)

Yes. The superbuild sets what it wants the prefix to be using -D on the command line. That creates a cache variable. set(CACHE) will respect that. set() without CACHE will shadow the cache.

[cobo (Cornelis Bockemühl)]

Thanks: adding some logic to the empirically found solution!

[ben.boeckel (Ben Boeckel (Kitware))]

Documented here:

When evaluating Variable References, CMake first searches the function call stack, if any, for a binding and then falls back to the binding in the current directory scope, if any. If a “set” binding is found, its value is used. If an “unset” binding is found, or no binding is found, CMake then searches for a cache entry. If a cache entry is found, its value is used. Otherwise, the variable reference evaluates to an empty string. The $CACHE{VAR} syntax can be used to do direct cache entry lookups.

[cobo (Cornelis Bockemühl)]

Ok, not so simple anyway! But I will add some comments to my scripts so I will remember.

Basically this means that the right “install” path is found not because the right one was in the project, but in spite of that setting - and this was possible because with the …CACHE… the sb was able to override it!

I was actually also not sure to what “${CMAKE_BINARY_DIR}/install” should evaluate:

• In the context of building just the project it is clear: setting CMAKE_INSTALL_PREFIX to that in the main CMakeLists.txt of the project avoids nonsense settings like the install path of the system (because this kind of “install” is most of the time not used for actual installations - at least as far as I would see it!). It makes sure that any install call without further specification would go to a subdirectory “install” within the target directory.

• In the context of the superbuild I assumed that it would evaluate to “<superbuild_target>/install” - which would be correct. And this because my understanding is that CMAKE_BINARY_DIR refers to the target directory of the “root project”, which would be the superbuild - while only CMAKE_CURRENT_BINARY_DIR would give me the result that I am actually seeing.

The reason must be that the superbuild “calls” the build of the projects not with add_subdirectory, but by calling another cmake instance, so making the project and not the superbuild the “root project” for that evaluation.

Ok, just “loud thinking” - seems to make sense now!

[ben.boeckel (Ben Boeckel (Kitware))]

@cobo

The reason must be that the superbuild “calls” the build of the projects not with add_subdirectory, but by calling another cmake instance, so making the project and not the superbuild the “root project” for that evaluation.

Yes. ExternalProject is used to build the subprojects. Most use CMake, but a few use autotools, at least one uses ftjam on Windows, and probably another one or two floating around. Almost none of them are set up to be used via add_subdirectory (conflicting global target names, assumptions about directory structure, etc.).