Problem with static mesh plugin

[luismhy (Luis Miguel)]

Hi everyone.

The other day I opened a post explaining a problem I have.

Spatio-temporal parallelism 5.11

Hello everyone. I would to ask about the “spatio-temporal parallelism” discussed in these two posts: https://www.kitware.com/spatio-temporal-parallelism-with-paraview/ https://www.kitware.com/paraview-spatio-temporal-parallelism-revised/ First of all, I explain my case. I am working in a project in which I have a big amount of heavy timesteps. Normally, Paraview process all these timesteps sequentially, taking around one minute for timestep. The result is that it takes approximately 8 or 10 …

My target is to optimize the post-processing time of heavy CFD datasets obtained by Fluent in format Ensight Gold. As a first workaround, I thought of performing this task in parallel, by using different processes to read separated timesteps. I will continue working in this, but, luckily, I have found another way of achieving my goal which could be really useful: the static mesh plugin.

https://gitlab.kitware.com/paraview/plugins/staticmeshplugin

https://www.kitware.com/staticmeshplugin/

Since my mesh does not change with the time and my files are written in Ensight Gold, it seems that this plugin would be perfect. Besides, despite its limitations, in many projects I only need to animate some slices, so there would not be any problem.

As per the requirements of this plugin, it is necessary to compile paraview using the flag “VTK_ALL_NEW_OBJECT_FACTORY=ON”, therefore the binary release cannot be used. For compiling paraview, I have followed the steps described in the paraview repository, for Ubuntu 20.

https://gitlab.kitware.com/paraview/paraview/blob/master/Documentation/dev/build.md

I have successfully compiled version 5.11.0 with this flag and I can run it with no errors, apparently. For the plugin, I have used the easy plugin builder, specifying the same version, 5.11.0.

https://gitlab.kitware.com/paraview/paraview-easy-plugin-builder

The plugin is also compiled with no errors. The problem comes when I try to load the plugin. It shows an error which says that the file “libmpi.so.12” cannot be found. Since I had compiled paraview with MPI enabled, I tried to compile it again with MPI disabled, to see if, at least, this error could be skipped. And yes, this error does not appear, but another does with another library: “libvtkFiltersParallelGeometry-pv5.11.so”.

I have noticed that these two files are not in the folders of my compiled version of paraview, but they exist in the official binary release for download from the web. Although the plugin should not work with the binary release due to the fact that “VTK_ALL_NEW_OBJECT_FACTORY” is not enabled, it can be loaded and used with this binary release. However, I have tried to read and Ensight Gold case and I cannot see any speedup by changing between timesteps, which is very strange considering the benchmarks described in page of the plugin.

So, at this point, there are two possibilities: or the plugin is not working well because paraview must be compiled with the previously mentioned flag (in which case I need help to solve the problem of missing libraries) or my Ensight files are not adequate for the plugin. Hopefully it is the first one.

Many thanks in advance.

[ben.boeckel (Ben Boeckel (Kitware))]

@luismhy

As per the requirements of this plugin, it is necessary to compile paraview using the flag “VTK_ALL_NEW_OBJECT_FACTORY=ON”, therefore the binary release cannot be used. For compiling paraview, I have followed the steps described in the paraview repository, for Ubuntu 20.

Ugh. I dislike that we have so many options that affect compatibility. Is it, perhaps, better if we instead make the specific classes support overrides explicitly? Which classes do you need to do this?

@luismhy

The problem comes when I try to load the plugin. It shows an error which says that the file “libmpi.so.12” cannot be found. Since I had compiled paraview with MPI enabled, I tried to compile it again with MPI disabled, to see if, at least, this error could be skipped. And yes, this error does not appear, but another does with another library: “libvtkFiltersParallelGeometry-pv5.11.so”.

Can you run from the command line with LD_DEBUG=libs in the environment? It might not be looking in the right places.

@luismhy

Although the plugin should not work with the binary release due to the fact that “VTK_ALL_NEW_OBJECT_FACTORY” is not enabled, it can be loaded and used with this binary release.

There’s no ABI incompatibility AFAIK here. It’s just that creating some types may not find the overrides you intend.

[luismhy (Luis Miguel)]

Thanks for the answer @ben.boeckel.

So, if there isn´t ABI incompatibility I need to use the plugin with a custom build with the flag enabled to see if it works for my cases.

I have executed paraview with LD_DEBUG=libs, both with my build and with the binary release downloaded from the web. The execution is up to the moment in which I load the plugin in both (and later close the software). The logs can be found attached.

I have compared the lib folders of the two installations, and I can see that many libraries which exist in the binary release from web are not in my build, so it seems that the problem is with my build, but I don’t know why. Is the build from source supposed to be exactly equal to the binary release from the web? Or is there a way to achieve it, perhaps with some specific options? The options which I have used for this proof are:

cmake -GNinja -VTK_ALL_NEW_OBJECT_FACTORY=ON -DPARAVIEW_USE_MPI=OFF -DPARAVIEW_USE_PYTHON=ON -DVTK_SMP_IMPLEMENTATION_TYPE=TBB -DCMAKE_BUILD_TYPE=Release …/paraview

ninja

Related with this problem, I have noticed an additional problem, but it isn’t critical. With the last version of CMake, the flag VTK_ALL_NEW_OBJECT_FACTORY=ON no longer exists, so it is necessary to use an older version in order to use it.

Many thanks.

log_binary_own_build.txt (209.3 KB)
log_binary_release.txt (468.8 KB)

[ben.boeckel (Ben Boeckel (Kitware))]

@luismhy

Is the build from source supposed to be exactly equal to the binary release from the web?

The superbuild provides a number of dependencies not typically present. Other than that, it is just a set of -D flags during configure as far as ParaView itself is concerned.

@luismhy

With the last version of CMake, the flag VTK_ALL_NEW_OBJECT_FACTORY=ON no longer exists, so it is necessary to use an older version in order to use it.

It is there; it is just hidden by default (it is, after all, a more advanced option).

Looking at the logs, the main differences seem to be that the package uses RPATH while your build uses RUNPATH. RPATH entries are transitively used, but RUNPATH (the new “rpath”) is not. You’ll need to provide -Wl,-rpath,/path/to/dir flags to find all dependencies when linking your plugin. See CMAKE_BUILD_RPATH and CMAKE_INSTALL_RPATH variables as needed.

[luismhy (Luis Miguel)]

Hi @ben.boeckel.

@ben.boeckel

It is there; it is just hidden by default (it is, after all, a more advanced option).

How can I enable this flag in the last version of CMake? I barely can find documentation of this specific flag. Probably I am not searching in the right place.

@ben.boeckel

Looking at the logs, the main differences seem to be that the package uses RPATH while your build uses RUNPATH. RPATH entries are transitively used, but RUNPATH (the new “rpath”) is not. You’ll need to provide -Wl,-rpath,/path/to/dir flags to find all dependencies when linking your plugin. See CMAKE_BUILD_RPATH and CMAKE_INSTALL_RPATH variables as needed.

Regarding this, there is something I am misunderstanding. I had already noticed (by reading the logs) that the package and my build are searching in different places, but, even if I modify this as you have mentioned, how is my build supposed to find these libraries if they don’t exist in any part of my system?

I can, of course, include the libs path of the downloaded version, but I am not very sure if this will work.

So, in my opinion, apart from that, I will need to rebuild Paraview with some specific options to get the libraries.

Many thanks in advance.