First own plugin integrated into custom app based on PV 5.7 RC2

Development
1

At the end of this post I come to a likely mistake in the current new RC2 build setup.

Before that I am just explaining some of the obstacles that a “user developer” may run into if he tries to work with the new build setup - just in case somebody is interested.

Since about a couple of days I am very busy trying to port a custom application with plugins to the new and much more modularized and transparent build setup that comes with PV 5.7 and which is now available as RC2. In other threads I was already asking questions, and this is now about the last step of integrating a first plugin into the custom application.

Generally speaking, a first problem with the entire setup is how to understand it at all in the first place! There are lots of automatisms in place, but once they do NOT immediately what you expect it takes a very long time to find a solution. The example custom applications and plugins are very helpful, but they do not show how to finally put things together.

So I first had to struggle with the problem of coming from a project like “clone1” - which is basically “Paraview with nothing”, i.e. with no plugins - to something real - loading the base and optional plugins of PV as far as they are needed.

Next I used the “elevationfilter” as a base to rework one of my own plugins. Building that ElevationFilter was not a problem at all. However, when added to the custom application, neither my own nor the ElevationFilter ever produced any shared library at all! Until after many hours of CMake code digging I found out that I need to set BUILD_SHARED_LIBS to ON because otherwise you will NEVER get any shared libraries for either modules or plugins! Looks trivial once you know it - but it is extremely hard to find out if it is not the case - and only this is the reason why I am telling this story at all!

With that it was possible to build the ElevationFilter plugin and my own within my custom project. They did not auto-load, but could both be loaded manually. However, while the ElevationFilter worked as advertised, my own crashed once it was supposed to do anything. The first assumption that it is some bug in my own code was after a while shown to be wrong: Somehow the “client server wrapping” did not work - but how the hell are you supposed to debug such a thing?

Again it was many hours of code digging and research until I realized that in the vtk.module file which is part of the new module system, the plugin MUST contain the line

DEPENDS VTK::FiltersCore

while in my own plugin I had tried to make it

PRIVATE_DEPENDS VTK::FiltersCore

…because I thought it might be good practice to have things “private” that are not supposed to be exported…

Finally the question: How to tell the custom application to auto-load the new plugins? Up to PV 5.6 this was done with some “magic macros”, but now this is supposed to work in a more elegant way - just adding the plugins to the paraview_add_client in a REQUIRED_PLUGINS section - and the custom application will try to load the named plugins - nice and much more elegant! (This is btw. also how loading optional plugins from PV is working). However - it did not happen.

Again after MANY hours of research I think now that there is actually a bug still in the new build setup system. The point is that with “logging” turned on I could see that my custom app actually TRIED to load the plugin, but it was looking for it in the wrong place - actually trying a long list of possible locations!

By default the plugin would end up in a subdirectory like

lib64/<PluginName>

However, such a location is not considered by the automatic loading! And also I would prefer if things are not so much “spread” around the place, but rather kept together. So told the paraview_build_plugin function that I want the libraries output into the same directory as the executables, which is in

bin

You can still specify a subdirectory there, but this I set to “” which is also the default anyway. But all this did not help: the plugins always ended up in subdirectories like

bin/<PluginName>

Which is clearly not what anybody would expect after the above specifications! And also the automatic plugin search never tries such a subdirectory.

Finally I found the place where that additional subdirectory level slipped in - and this is what I think is probably a mistake: the paraview_add_plugin inserts this subdirectory level for no obvious reason in two places, so I had to delete (or comment out) the following two lines:

CMake/ParaViewPlugin.cmake(593)
CMake/ParaViewPlugin.cmake(932)

With this everything worked fine finally!

I have no idea if this additional subdirectory level may in some situations be important to avoid clashes or whatever, but then the problem would have to be solved at the level of the auto-loading mechanism - in a way that the plugins can be finally found!

2

Sorry for all the frustration; documenting all of this for downstream usage is on my plate for 5.8. If you could please file an issue with the pain points you hit so that I can make sure that those get attention, that’d be really helpful. Getting plugins to be found automagically is now done on master by using the plugin target and a qt.conf-like mechanism for finding the plugin XML files associated with those plugin targets. The plugins are then located relative to that XML file as well.

As for the logic behind the subdirectory: modules the plugin builds privately are also placed beside it. This allows the plugin directory to be a single unit rather than having a bunch of plugin libraries living beside plugin-private module libraries as well. A lot of ParaView plugins have these, so you can see the rationale behind the subdirectory more there.

3

Thanks for answering after your hopefully enjoyable vacation!

Actually I am already at further construction sites - and it is not getting easier! But by now I am happy that customers are currently not asking too much while this “porting 5.6 to 5.7” project is totally running out of control (I supposed it to be a project for one-two weeks but now it is almost 3 months and not finished yet…). The positive side is of course that you always learn a lot in such circumstances…

Basically I already wrote this post on discourse with the intention to help others shorten this struggling time - and also for myself as a reminder! It contains already a number of “pain points”, but others may run into others of course - depending on their “initial assumptions” about how things should work - and then not do.

Regarding the loading of plugins, I am doing it now “manually” anyway in the constructor of the main window of my custom application. The reason why I am not using the elegant new setup is simply the fact that in the same constructor I also change the standard main view to be something else than the standard render view, but my replacement is loaded as a plugin - and it will happen too late if I am using the standard mechanism! And in my custom loading function I already take account of that extra directory level.

However, this is not applicable to most other projects, so making sure that the plugins are found in the right place, in whatever way, is certainly a good idea! But I cannot say if and where there is any possible bug - since I also see that RC2 is now not any more the current state of the art. (I switched to the “official release” of v5.7 by now).

Btw, my current construction site is the problem of having one custom application, then another one “built on top”, using a few more extra plugins. And just today I found out that all my attempts to do this with install(TARGETS…) and install(EXPORT…) are basically nonsense: it is much more convenient and appropriate to do it with export(…) directly! Because there is next the “superbuild” that also needs to run - and that already needs a proper install(TARGETS…) setup in the project. But I guess I am about to find my way - gradually!

You see that all this is of course not Paraview specific, but CMake stuff, but one goes into the other…

For all these things for me the major pain point is the fact that there is a lot of docs available, but all only useful once you would have understood basic concepts! Like already: what means “install” in a world where nobody really installs software anymore with “make build” and “make install”. So “installing” in that sense is obviously obsolete! Not in a CMake world, because CMake uses the term “install” now for many other useful things - but also finding this out was a very hard and steep learning path! Only once you understood it looks sometimes even trivial…

Bottom line: at least for people with a mindset like mine, explaining concepts in a “human language” is much more important than explaining details in a purely “techie language” - with “interfaces”, “modules”, “kits”, “targets” etc. without clearly explaining what these things actually mean in a certain context. This of course only as a complement to the already very well developed docs for the single functions, parameters, variables, commands etc. in the CMake world!

4

On master, there is now a signal you can listen to which will happen after the plugins are loaded. This is where you can do anything that assumes they’re available.

As mentioned in the other thread, I’ll take these pains into consideration for the docs. Do you have a gitlab account I can ping when the MR is up (should be within a month)?