PV Custom application version 5.7: Where are the PV plugins?

cobo · August 24, 2019, 1:37pm

Right now I am trying to port my PV “custom project” from being based on PV 5.6 to 5.7 - not the least because I like the improvements in project setup, with more modularity and transparency. And just because of that I decided to simply start again 1) with an “empty clone project” (Clone1), then 2) add all the required PV functionalities and finally 3) my own extensions: The code is already there and should mostly work, but the project setup should better be redone from scratch.

Right now I am at 2) - and it looks like I am getting some kind of “almost Paraview” so far - only that there is not one single Plugin loaded by default! If I remember well, this was not the case when I did that kind of exercise with an earlier PV version.

Actually it’s not so bad: I find at least many of them if I look into the “Plugins Manager” where they are listed as “not loaded”. Only what I do not see there are the “very basic” plugins, like the cutter or slicer for objects etc.

So I have now a question and a suggestion:

QUESTION: Is there anything I have to do now in addition to what was required in earlier versions? I mean: the entire way how the plugins are grouped in “modules” etc. is new, so I would not be too much astonished.

SUGGESTION: The Clone1 project is good and nice - a very much stripped down PV with no filters at all. However, I could imagine that sometimes another “Clone…” example could be useful that is more or less the opposite: a custom application that starts with a “full blown” Paraview, with the settings and options like they are specified in the imported (keyword: find_package(Paraview)) Paraview project. The current Clone1 would then be for people who want to use the “Paraview base” for a completely different and much more focussed application. But the other one would be for projects that are more like “Paraview plus something extra”. And it seems that starting just with a renamed ParaViewMainWindow does not work: that would then rather be a branch of the Paraview project, not a “custom application”.

mwestphal · August 26, 2019, 7:37am

Plugin should be enabled by default during the superbuild, You can check in the superbuild/paraview/build with ccmake if the plugins have been enabled. And also run the actual ParaView executable to see if they are available.

I just checked, Clone1 is a full blown paraview, not a stripped down ParaView. The only things it seem to miss are the plugins, as they are not built by Clone1.

cobo · August 26, 2019, 8:29am

Thank you Mathieu!

I think I was not clear enough: I was asking only about the auto-loading feature! So let me explain more precisely.

Yes, I understand that Clone1 is a “full blown Paraview”, and also I understand that I have to enable plugins in my “base Paraview” in the first place. This includes not only things like the GDAL plugins that I am using, but also enabling Python etc. And all this I did properly: In my self-compiled PV 5.7.RC2 all this is working perfectly!

I am not at the “superbuild” yet - just trying to get my AthosGEO “View” (which is supposed to be the OS version) built and running step by step. And the first step was: start with Clone1, setup a project for it that “includes” that self-compiled PV 5.7 (with find_package…), then build and run. What I would like to see then is a program that is basically PV 5.7, only that the title bar says “AthosGEO View”. Not very ambitious, but I am trying “step by step”.

But what I am getting now is this:

This is basically PV 5.7 with a “AthosGEO VIew” title bar, but without ANY plugins. So like what you were saying about “Clone1”. If you call this “full blown PV” is now a question of wording. So what I actually mean is: a full blown PV with all the plugins auto-loaded that are also loaded in my self-compiled PV 5.7 - after setting the proper build options.

If I am looking into the plugins manager, this is what I see:

Thus all the optional plugins are there, but not loaded. I can now go and load all the list, and then this is what I am getting in the “Filters - Alphabetical” menu:

However, I am having all the optional plugins now, but still none of the real “basic filters”, like “slice”, “cut”, “extact selection” etc. etc.:

So my question is still:

Where are these filters?
How can I trigger the “autoload magic” in such a way that I can load specific filters already immediately after the build! Of course I know that I can switch on the “auto load” checkbox in the plugin manager, but I am talking about some kind of “default auto-load” that is already done before the user even knows anything about the plugins manager.

I am not really sure that these things were working differently in 5.6 than in 5.7, but in any case, at this very moment I cannot get any plugins automatically loaded, and I do not see the “most basic” filters at all!

mwestphal · August 26, 2019, 8:36am

Regarding autoloading the plugins, this has been answered here :

Regarding the missing filters, they should be here and they are here in my Clone1 build. Do you have the same behavior with the default Clone1 application ?

cobo · August 26, 2019, 8:39am

Thanks for the pointer to the existing answer (thus indeed something has changed there!), and regarding the second - I will have to check again (because I was already at the point to compile “naked Clone1” and did not really pay attention). And then I will possibly/hopefully also find the answer.

cobo · August 26, 2019, 9:02am

…still I am not fully satisfied with that “solution” that is offered for the autoload question: All the three locations where I can put hands on the plugin loading process are not available at build time, but only once the software is already there!

Ok, for the superbuild you would have to introduce some funny magic to make it happen on the user computer during installation, but if I am about to build a PV derivative on my own computer, I have to do things completely outside of the build setup in order to get it happen.

In other words: I will probably find some workaround for myself, but overall I think that the removed PARAVIEW_AUTOLOAD_PLUGIN_* CMake options are leaving a gap that is not closed by the alternative ways to trigger autoloading - because the options were part of the build setup, while the others are not.

mwestphal · August 26, 2019, 9:11am

I definitely agree, that’s why I opened an issue :
https://gitlab.kitware.com/paraview/paraview/issues/19276

cobo · August 26, 2019, 10:28am

It turned out that the disappearing “basic plugins” were due to the fact that I went a bit too far with renaming: I renamed the ParaViewFilters.xml (and similar) files to AthosGeoViewFilters.xml etc. - which is fine as long as I apply the same change also in the CMakeLists.txt file in the paraview_client_add call.

However, also inside the file I changed the tag in the same way - and this is obviously not recognized by the server manager!

Just trying to add now also the “GeodesicMeasurement” filters to that XML tag, into some own submenu etc. I was asking myself whether this would already trigger an auto-load of the corresponsing plugin, but this is not the case (like the experts would have told me from the beginning…).

But after I loaded the GeodesicMeasurement plugin through the plugin manager, that new submenu appeared nicely, with the two filters that are part of the GeodesicMeasurement plugin.

My next idea was to load the GeodesicMeasurement plugin like I always used to auto-load my own plugins - with the PV_PLUGIN_IMPORT_INIT macro in the header part of the cxx file of the main window class, and the PV_PLUGIN_IMPORT between the menu and the behaviour setup in the constructor.

However, it turned out that this macro is not defined, although vtPVPlugin.h is imported. BUT other than in earlier versions, the macros are defined between #ifdef PARAVIEW_BUILDING_PLUGIN … #endif - and that seems not to be defined! In version 5.6, this #ifdef was not there yet.

Which again raises two points:

a) for the purpose of autoloading the GeodesicMeasurement plugin it is probably the wrong approach anyway - because the plugin is already built (in the PV build), and it exists as a shared library, so no way to afterwards still linking it statically: should not work actually if I am thinking about it!

b) however, later on for my own plugins, I will eventually run into the problem again…

cobo · August 26, 2019, 2:15pm

NOTE: I edited this post twice, so if you read an earlier version…

Workaround that is ugly but seems to work:

in the CMakeLists.txt file:

#we need Paraview here for the dependencies
find_package(ParaView REQUIRED)
#make sure AthosGeo can find optional ParaView plugins for explicit loading
configure_file(
“${CMAKE_CURRENT_SOURCE_DIR}/ParaViewPluginPath.h.in”
“${CMAKE_CURRENT_BINARY_DIR}/ParaViewPluginPath.h”)
the ParaViewPluginPath.h.in has the following content:

#define PV_PATH_STRINGIFY(x) #x
#define PV_PATH_TOSTRING(x) PV_PATH_STRINGIFY(x)
#define PARAVIEW_PLUGIN_PATH @ParaView_DIR@/lib64/@PARAVIEW_PLUGIN_SUBDIR@
#define PARAVIEW_PLUGIN_PATH_STR PV_PATH_TOSTRING(PARAVIEW_PLUGIN_PATH)
and finally at the end of the main window constructor I put the following code:

// explicitly load the GeodesicMeasurement plugin from ParaView
pqPluginManager* plgmgr = pqApplicationCore::instance()->getPluginManager();
std::string pluginLib = std::string(PARAVIEW_PLUGIN_PATH_STR) + “/GeodesicMeasurement/GeodesicMeasurement.so”;
QString err;
pqPluginManager::LoadStatus res = plgmgr->loadExtension(nullptr, pluginLib.c_str(),
&err, false);

With this I am getting the GeodesicMeasurement auto-loaded - just as an example. Of course this is only working with Linux so far (lib64 subdirectory and .so extension!), but a Windows version should be feasible as well - with some more ugly #if… code.

cobo · August 27, 2019, 7:58am

Easier solution - without the need to pass the Paraview directory from the CMake setup into the program (which is btw not working on the computer of a customer - because he will get the PV build path on the computer of the programmer…):

std::string plugin_xml =  "<Plugins> <Plugin name=\"GeodesicMeasurement\"  auto_load=\"1\" /> </Plugins>";
vtkSMProxyManager::GetProxyManager()->GetPluginManager()->LoadPluginConfigurationXMLFromString(plugin_xml.c_str(), nullptr, false);

This leaves the work of finding the plugins to the responsible instances within PV (ie, the plugin manager) and works with only the name - which should thus also be independent from the operating system and running environment at the customer’s site.

ben.boeckel · August 27, 2019, 12:42pm

So the plugin stuff has been vastly improved on master the past few weeks. If you pass PLUGINS_TARGETS ParaView::paraview_plugins to the paraview_client_add function, your application will find ParaView plugins by default. However, ParaView still controls the AUTOLOAD status of them (which is currently failing due to the issue mentioned above). However, the plugins are now loadable by name since they’ve been found. You should be able to add any plugins you want to autoload in your application through the REQUIRED_PLUGINS argument to paraview_client_add. That makes the XML you’re referencing internally and does the same thing.

cobo · August 27, 2019, 4:34pm

Thanks for the hint: It mostly worked for me - with my RC2 release that is two weeks old!

This is what I tried:

REQUIRED_PLUGINS
    AcceleratedAlgorithms
    ArrowGlyph
    BagPlotViewsAndFilters,
    GeodeticMeasurement,
    EyeDomeLighting,
    PanoramicProjectionView,
    StreamLinesRepresentation)

Actually auto-loaded were AcceleratedAlgorithms, ArrowGlyph and StreamLinesRepresentation, while the others where present, but still had to be loaded manually.

The PLUGINS_TARGET with ParaView::paraview_plugins did not work, and to me it looks like such a target is not being defined by find_package(ParaView). At least this is what I conclude from the error message:

Target "AthosGeoView" links to target "ParaView::paraview_plugins" but the
target was not found.  Perhaps a find_package() call is missing for an
IMPORTED target, or an ALIAS target is missing?

Of course it is never really clear for the “non-initiated” which targets are available at a specific point and which not, but in this specific case I was first of all looking into ParaView_targets.cmake that is generated somewhere in the build target tree during the build of ParaView. There a list _expectedTargets is being initialized, and this does a lot of targets, but not any ParaView::paraview_plugins in my case.

Right at this moment I also downloaded the “master of the day”, and it is on the way to compile. Anyway, already I can say that the paraview_plugins target did not “appear” in that newer release.

ben.boeckel · August 27, 2019, 5:42pm

The , (comma) you have on some of those would be a problem . The PLUGINS_TARGETS magic is only on master (it was too invasive to the plugin system to backport to 5.7).

cobo · August 28, 2019, 6:31am

The commas were the problem indeed - THANKS! (commas are C++, not CMake…) Much easier solution than moving ahead to “master” and trying whether this would bring the solution…

So I will stay with 5.7-RC2 (and eventually switch to the final release of 5.7), because basically I feel better if I take the train where it is standing in the station, not jumping in while it is running.

Next problem to solve: How do I get the “Reader, Filter and Writer Reference” into my custom application? So far I get only help for those filters that are loaded “later on” (ie the ones that are loaded with the REQUIRED_PLUGINS - like Geodesic Measurements), but not the “base items”. Instead in the “Output Messages” dock window it says “Could not locate index.html”.

But this is already a next question - and most likely I will have to find another stupid mistake…

ben.boeckel · August 28, 2019, 11:28am

@utkarsh.ayachit Thoughts?

cobo · August 28, 2019, 12:09pm

Actually I found some solution: just include the existing paraview.qch file in the CMakeLists.txt file:

# access to the ParaView documentation
set(paraview_qch "${ParaView_DIR}/Applications/ParaView/Documentation/paraview.qch")
paraview_client_add(
    NAME AthosGeoView
    ...
    QCH_FILE
        ${paraview_qch}
    REQUIRED_PLUGINS
        AcceleratedAlgorithms
        ArrowGlyph
        BagPlotViewsAndFilters
        GeodeticMeasurement
        EyeDomeLighting
        PanoramicProjectionView
        StreamLinesRepresentation)

The result is so far ok for me:

Which means that I have now exactly the ParaView help from the underlying ParaView, plus separate Help sections from those optional plugins that are providing them.

Later I will have to add another Help section - for my own extensions! And for that I will try to find out how that paraview.pch is generated, and then add it to the Help system - ideally in such a way that it would come before the ParaView help. A test showed me that I can actually have more than one .qch files in that QCH_FILE section: I included a CMake manual first, and result was indeed that I had now first a CMake manual, then the ParaView manual, and then again the optional plugins documentation.

Maybe there are more elegant solutions (that would e.g. not require the relative path of the paraview.qch file below the ParaView_DIR path), but at least it works now!

ben.boeckel · August 28, 2019, 9:42pm

This MR should make it clearer that multiple QCH files can be specified: https://gitlab.kitware.com/paraview/paraview/merge_requests/3540. I think it was a mismatch between the API input towards what happened towards the implementation.

set(paraview_qch "${ParaView_DIR}/Applications/ParaView/Documentation/paraview.qch")

Well, this works for a build tree, but will fail with an install tree. I’ve filed an issue here: https://gitlab.kitware.com/paraview/paraview/issues/19284

cobo · August 29, 2019, 7:06am

Thanks for the comments and filing the issue!

Making it clearer would be implicit if the tag would not be QCH_FILE but rather QCH_FILES - with an S at the end.

More generally speaking I would say that it makes a lot of sense to also include the ParaView filters/readers/writers… docs after including the plugins themselves into a custom application. And to make this possible, for me the most obvious way would be to make some “target” available after the find_package(ParaView) call, like ParaView::plugin_help or something, that could be added with the target_link_libraries. Which would basically assume that it exists already as a library.

For later on generating a similar help document for my own plugins I studied the generation process in ParaView - and of course I do not understand all the details yet! But the steps are as follows:

collect a list of all the server manager XMLs that contain documentation “snippets”
with some XSL transformation, extract the docs and convert it all into HTML which together with some additional files (index.html, table of content, CSS etc.) make up the help document
collect all the HTML and other files into one single QCH document
add this to a resource file (QRC) and use the Qt “resource tool” to convert this into some C++ with a lot of explicit binary stuff
compile and link into the application where the document can be accessed like a “file” with a “virtual path” starting with a “:”

My first attempt was to redo this entire process in my custom application and use the functionalities and macros for this purpose that are already existing in the Paraview build system. However this approach failed already with the fact that I did not get access to the list of server manager XMLs.

Ok, such a list could be “exported” through the find_package mechanism, but then still a lot remains to do because many of the ParaView build macros and functions are still “hidden” and would have to be cloned - and I do not know how much code I would have to “pull out” and duplicate for this to happen.

An advantage would be that it opens up the option to add help docs for own extensions directly into the original document and adapt it in all possible ways.

The approach that I finally realized is somewhat “hackish” - and of course I hate the fact that I am learning through the remark of Ben Boeckel that it would not work for installation: I was so far thinking that the QCH would not “live” any more in the installation tree anyway, being linked into the executable as a “resource”, but as I said initially: There are still many things that I do not understand yet!

The most comfortable but least flexible approach would be to make the plugin help doc available as a “target” that can simply be added with some target_link_libraries call and probably still an initialization function call in the main window constructor.

ben.boeckel · August 29, 2019, 12:48pm

Making it clearer would be implicit if the tag would not be QCH_FILE but rather QCH_FILES - with an S at the end.

That’s what the linked MR does .

many of the ParaView build macros and functions are still “hidden” and would have to be cloned

Which ones? ParaView shouldn’t have anything private other than helpers (of which I don’t think any exist).

The most comfortable but least flexible approach would be to make the plugin help doc available as a “target” that can simply be added with some target_link_libraries call and probably still an initialization function call in the main window constructor.

Agreed. However, it will likely not make it until 5.8 (I’m headed out on vacation for a few weeks soon). I think getting projects to export their documentation will need some API design work and thought.

cobo · August 29, 2019, 1:11pm

If that is the case - all the better: then it is “only” a question of understanding - which basically means “code digging”…

Because regarding my own project, I would actually prefer to generate one single documentation, following a bit the setup that I find in Applications/ParaView/Documentation/CMakeLists.txt. I think that I would somehow get this together - except that there is one variable that is available inside the ParaView build, but not inside a custom application:

paraview_server_manager_files

In that above mentioned CMakeLists.txt it can be assumed (and I tested it) that it contains a list of all the server manager XML files - like the naming says it already. These are somehow “collected” as a side effect of a number of other calls: no idea how I can get this variable out of the Paraview project build setup and into my own!

Btw and off topic here: I found a little bug in the ParaView setup files that is related to handling a splash screen image: in ParaViewClient.cmake(207) [referring to RC2] I had to change this line

  ":/${_paraview_client_NAME}/${_paraview_client_splash_image_name}")

into this:

  ":/${_paraview_client_NAME}/${_paraview_client_splash_base_name}")

Without this change, the pqInitializer::Initialize() function would look for a resource named <…>_splash.img while in reality the name of the resource would be <—>_splash - without the “.img” extension. (Of course the fix could have been done in a way that the naming of the splash bitmap is adapted to have the extension - only it has to match!)