Since weeks now I am struggling with the fact that since some time, my Superbuild for a ParaView custom application does not properly include the Python runtime any more. All is done on Windows 10, because that is what my customers are using. This happened when I switched from underlying 5.8.0 to 5.9.1, but I also did changes in my own code.
After a lot of research with versions when it still worked and the current one, I found out the following:
A functional Python3 runtime in an installed software includes c:\Program Files\MySoftware\bin\python37.dll (or whatever version), plus subdirectories named Lib and DLLs with “a lot” of content. A non-functional Python3 runtime has only the dll plus a Lib with not much content - to put it a bit simple. Symptom when trying to do something in the Python shell in the “new versions”: Click into the shell - and the program crashes. It does so inside the python.dll when trying to initialize: Looks like it is looking for that missing stuff. If I add it manually into a buggy installation, the Python subsystem works. This was the state when I realized that something is going wrong - and I started to check and manipulate the different configuration files until I came “one step further” - as follows:
In my superbuild target directory, there is a subfolder named /install. If I choose to not use a local Python system, some source code of Python 3.8 is downloaded and built, and the result ends up in a subfolder of this /install folder: /install/Python. This contains python.exe and dll, plus the mentioned subdirectories and many more. It is possible to run Pythin from a command line inside this folder, so this seems to be a fully functional Python system. To me it looks like it should eventually end up during the ctest -R … call all inside the _CPack_Packages/win64/NSIS/mysoftware/bin folder, but this does not happen. Instead I am right now getting a complaint during the CTest… run that some specific paraview dll does not find the python.dll that it relies on. The search path would include the install/bin subdirectory, so if the Python system was there and not in install/Python, it would be found. If I try the same using a functional local Python system (from OSGeo4W), I have there such a Python folder with the necessary subfolders, and I assume that somehow that stuff should be copied from there.
Bottom line: I do not get any further at this moment!
My assumption is that somehow the packaging process should FIRST somehow put the entire selected Python system to the destination folder (including dll and required subdirectories), either from the self-compiled output or from a local source, and only then try to put also any other stuff that relies on an existing python.dll. Or else, just copy that python.dll to the install/bin folder, to make that other script happy, and only then put the entire Python system into the package.
I hope somebody is able to figure out what my problem is without too much trouble, because that superbuild is pretty complex and I do not believe that there are lots of people around who understand it in such detail that they can easily answer my question! still I would be happy if somebody has some helpful hint.
Before running ctest -R …, I added the paths to the python39.dll and some others that I am using from my own computer. With this, the packing ctest call was running through. It is also possible to install the generated package successfully. However, if I click into the python shell of the installed program, the program crashes without any message.
In the attached log ctest.log (385.0 KB)
it can be seen, that python39.dll was copied even several times to different target locations (exactly 9 times), but NOT the related Lib and DLLs folders. No idea what the purpose of all these different python39.dll files is, but the important one is of course in C:\Program Files…\bin.
Guess: There is a mechanism within this package generator that “pulls in” those DLLs that are needed by the ones that are explicitly packed. For this, the PATH is being used, and with this automatism, python39.dll is added to all those target folders where it is needed by some other dll.
If now I manually copy the Lib and the DLLs subfolder from C:\OSGeo4W\apps\Python (my source for Python) to C:\Program Files…\bin - and now my program is working correctly: I can click into the Python shell, triggering the initialization of the Python system, and work with the Python shell. In the About dialog I can see that the used Python system is indeed the one that I installed and manually completed.
My problem is now: How do I convince that superbuild to include these missing Lib and DLLs subfolders automatically?
During further research I still did not come up with any answer to my question. That is basically: How is the Python inclusion supposed to work in the ParaView or derived superbuilds? It worked for me with 5.8.0, but not any more with 5.9.1, but I can also not simply blame these versions because I also did a lot of changes myself.
In the first case I did not understand why it is working, but I was glad it was. Now I still do not understand, but it does not work - and this is more serious…
Anyway, I went now for a workaround. After finding out that having the right DLLs and Lib subdirectories does the trick to make an embedded Python system work, I simply added two more “projects” to my adapted custom superbuild, both only copying files: one for the DLLs folder, and one for the Lib folder. This works if I am having a local existing Python system that I can copy and which cooperates also with my ParaVIew custom application build, in terms of MS Tool Set.
I would still be interested in an answer to my original question, which may be a bit tricky for most except the one who wrote that Python stuff in the superbuild! But for the moment I am glad I have a “somehow” working solution.
No idea how useful it is this late, but I can’t say I really remember the details that well either (it’s all in fixup_python.*.cmake), but I have no idea how well the superbuild’s packaging works with a Python it did not build itself. It’s certainly never been well-tested.
I don’t really see much in the 5.8 to 5.9 diff besides some Apple support and removal of Python2 support though… Maybe something there fell through the cracks (e.g., python_enabled went away)?
Right now I am working with that mentioned workaround solution. In the “superbuild” sources for my custom project, I anyway added some .cmake files for my own extenstions (like athosgeoview.cmake, athosgeoblend.cmake, athosgeoblendtutorialdata.cmake and so on) in the \projects subdirectory. In the same way I added now two more such project inclusion files: athosgeopythondlls.cmake and athosgeopythonlib.cmake, because the \dlls and the \LIB subdirectories are the ones that are actually “forgotten”.
That works for me, but it would not be a general solution for a ParaView superbuild, and it does not care for any options like including either self-built Python or an existing one etc.
Thanks for your hints anyway: You are giving me hints for eventually having another look into the issue - always keeping in mind the “old wisdom”: Never change a winning team!
