Modules and kits - what is the difference?

Looking through the reorganized build system of ParaView 5.7 RC2, I find modules (with specification files vtk.module) and kits (specified in vtk.kit files) - and the question arises about what is the difference? Because both seem to serve the purpose of “modularization” of the complex software base. Even with some searching in the internet and reading announcements and explanations here and there, I did not find any clear explanation about what is their specific use; I only have kind of working hypothesis:

a) There is some kind of hierarchy, like one “kit” can contain several “modules” - or maybe also the other way round!?

b) It’s only historical: “kits” are an earlier attempt to modularize, but then a better approach was found - and in order to clearly distinguish it, the new units were called “modules”. Meaning that “kits” would eventually disappear - and new ones should not be generated.

c) There is some kind of functional difference, like “modules” can be imported into other projects, while “kits”… - actually no idea…

The fact that in some folders you find both a vtk.module and a vtk.kit does not really clarify the issue for me.

Background of the question is the intention to follow the current best practice also for an own project, but for this it would be best to first understand it a bit better!

Kits contain modules. They’re an implementation detail in order to reduce the number of resulting libraries. We’ve had projects which have had too many load commands (each library needs one) on macOS, so kits were envisioned as a way of reducing that number.

The new build system got rid of lots of cruft; if they were unnecessary, I’d have not implemented them. They were quite tricky to get right.

Kits are an implementation detail of the project using them. Whether kits is enabled or not, the CMake target to use is VTK::CommonCore even though in a kit build, the functions end up in a vtkCommon library.

The kit file is placed in the “lowest common denominator” module of a kit. Usually some FooCore module in VTK or ParaView.

The documentation for kits is pretty thin. If you have problems with the number of libraries in your project, using them can help. If not, I wouldn’t worry about it too much. What modules got into what kits is a bit of an art; cycle detection is left to CMake to figure out at the end, but the output usually helps finding the culprit. Here’s some docs, but it could certainly be expanded:

https://gitlab.kitware.com/vtk/vtk/blob/5345571164a62176469f5a38507d2e9e366ccf4f/CMake/vtkModule.cmake#L234

The main relevant bit:

Kits are intended
to bring together related modules into a single library in order to reduce the
number of objects that linkers need to deal with.
1 Like

Thanks a lot for the clarification!

This morning I just tentatively sorted my source code into potential “modules” - and it looks like it would be about 50, including plugins, utilities, views, representations, one “manager” and much more. (Plus I found 6 that are actually dead code at the moment: without that detangling exercise I would not have realized this, so the modularization already pays off!) That’s not yet a nightmare, but enough to get lost already a bit!

From your words I conclude that I am ok if I start making all these potential modules real modules and try to get it compiling and running. And leave kits for a potential next step of cleaning up the code.

The project will also be the base for further projects that import it completely and add more functionality. Like my custom application already does it with Paraview. If I get it right, kits could then also be a way to keep things together that would normally not be separated anyway and end up with shorter lists of dependencies that way. So exactly in the sense of your last sentence.

Finally - the custom application COMPILES together with one single module that includes one plugin - and I am happy that I finally managed to get so far with the help from Ben Boeckel (and a lot of code digging)!

Learnings:

  • How to find the required modules to link if there is a specific #include <vtkWhatever.h> not found - using grep and finding the right CMakeLists.txt file in the ParaView sources: Tedious but generates a lot of transparency that was not there before

  • Also it is now clear for me that a better way to generate “modules” is to put many more classes into one - not only “just a few” - just following the example of VTK and ParaView. Which means that at the end I will NOT have 50 modules but many less

Still not there: while both the executable as the plugin do compile, I still do not get a shared library for the module, nor a static one that would be linked to the executable! (Or rather: if the static lib that I am getting WOULD be linked to the executable, I still do not see any trace of it…)

In the “good old days of 5.6”, my build ended with an executable plus plugins as libNNN.so (shared libraries) that would be auto-loaded in the main window constructor using PV_PLUGIN_IMPORT. Now I neither have a libNNN.so (but only a NNN.a - with NNN being the module name), and PV_PLUGIN_IMPORT is disabled by some “protective #ifdef” that is only defining it if PARAVIEW_BUILDING_PLUGIN is defined - but no: I am building not just a plugin, but an entire custom application with an executable PLUS plugins.

So still two questions open to finally feel like I am “on track” for the transformation:

  • How do I finally get a shared library from my “module” that I can load, eg. manually with the Plugins manager?

  • What is the way to automatically load the plugin once it exits as shared library if not any more PV_PLUGIN_IMPORT: Is it just about adding it to the REQUIRED_PLUGINS of the paraview_client_add - like I did successfully with the optional ParaView plugins?

How to find the required modules to link if there is a specific #include <vtkWhatever.h> not found

See Utilities/Maintenance/FindNeededModules.py in VTK for this. It’s sort of VTK-specific on its actual output, but ends up getting you what you need.

How do I finally get a shared library from my “module” that I can load, eg. manually with the Plugins manager?

Using paraview.plugin files and such. You can use FORCE_STATIC in the paraview_add_plugin call to make a static plugin. This will be initialized when the paraview_plugin_build target is used from a client via PLUGINS_TARGETS.

What is the way to automatically load the plugin once it exits as shared library if not any more

Either passing the plugin name to paraview_plugin_build(AUTOLOAD) (and then using the .conf.xml autoloading mechanisms; master/5.8 only) or paraview_client_add(REQUIRES_PLUGINS) (also works with 5.7).

The very “simple” problem is: I can do whatever I want - I do not get any *.so file at all - neither for the module nor the plugin or anything! So there is simply nothing that can be loaded, whether “auto” or not.

So I just tried to simply build the “ElevationFilter” example from the Paraview 5.7-RC2 source code as a completely separate project. That works like a charm, but I see that it is exactly the same there - no *.so file whatsoever is built! So no idea how to finally get such a shared library at the moment.

If I do a find -name *.so in the build tree of PV5.7 itself there are tons of such files that look like they are modules at first sight… Maybe I have to simply forget that “ElevationFilter” for the moment which I took as my “teacher” and try to find out how the trick was done with some of the modules that I find in PV5.7!?

After a thorough study of the paraview_add_plugin function, and throwing some message() stuff in, I finally found the missing part: somewhere BUILD_SHARED_LIBS has to be defined! Otherwise you can stand on your head and you will NEVER get any shared library…

Ok, that imported plugin of mine still crashes (which it did not do before), but that looks like “normal debugging” - and I will peacefully go no into some other weekend activities!

Once more THANKS to Ben Boeckel for guiding me along so many cliffs and obstacles through this week!