Solved once - back again: The vicious "unable to find api-ms-win..." error message

Development
1

It’s again about the vicious failure of installation package creation after paraview (custom application) superbuild:

RuntimeError: Unable to find the api-ms-win-crt-runtime-l1-1-0.dll library from […]

Thanks to some patches of Utkarsh in some of the cmake scripts I could generate package successfully, but now on another computer with some changed setup etc., I ended there again. Right, some of the patches were not completely there: apply them - try again - still same thing… (not exactly: this time it is another one of these api-ms-win- dlls that is missing)

Ok, I did not fully clean out the entire superbuild because that takes many hours on my rather slow system where I am testing this. And I am sure that it would be basically enough to simply delete a few files or folders in the target directory of the superbuild to avoid recompiling entire paraview (which is basically the thing that takes the time) and get the fixes… in place.

My problem: I simply do not fully understand the “magic”!

What I understand: this entire thing is again some kind of “fighting against Microsoft” because some years ago they introduced these api-ms-win…dlls are somehow “special” and maybe should not even be put into any software packages (!?)

Actually the superbuild script is searching the PATH for that missing dll, and does not find it. However, if I am using “find” from the bash shell I find that file name 26 times on my computer! For example in:

C:\Windows\System32\downlevel\ -> size 16288 bytes
C:\Windows\SysWOW64\downlevel\ -> size 15776 bytes

So obviously they are not identical and I should not simply add one of these many locations to my PATH in order to “solve” that issue: I may make it even worse!

On the other hand I do not directly see the “crucial code” in the Utkarsh patches where e.g. dlls with the name pattern api-ms-win* would simply be skipped.

To finish: If there is somebody who could give me a few hints regarding what is going on it could save me possibly many hours of trying to rebuild - rebuild - rebuild - and still not fully hit the “crucial point” that makes it happen! Because it seems to be a recurring issue, and just some very little error would bring it back.

(For the rest I have a lot of respect for these Kitware people who do this kind of struggle for days, months and years, so I also do not want to “steal” too much of their valuable time with my tricky questions: You could just as well say: “Clear everything, restart your superbuild including entire paraview rebuild, see if it works, and otherwise try again: It SHOULD work because it already did in our place!” And I know the learning effect would possibly be even higher…)

2

the key is fixup_bundle.windows.py. We should make it so that its skips looking for and installing files like api-ms-win-… etc. First determine which dll/exe is causing it to look for the offending dll and then updating the checks so that it skips the offending dlls would be the trick.

3

Ok thanks, so you are not directly filtering for the api-ms-win… files, but for their “parents”, which would typically be some Windows related dlls as far as I understand. And this seems to happen in the “dependencies” function (line 132ff):

line 139: system_dlls = re.compile(r’[a-z]:\windows\system..dll’, re.IGNORECASE)
line 145: msvc_runtimes = re.compile('MSVC[A-Z][0-9].dll’)

So next time I will know where to check - and for the moment I am having a complete rerun going - with exactly these lines in exactly that file, so it SHOULD succeed this time!

4

The packaging test logs all of the links it encounters (see Testing/Temporary/LastTests.log or run with ctest -VV), so finding the binaries which need it shouldn’t be too hard.

Also note that the cpack- tests are independent of the build; they only look in the install/ directory for what to use (though rpath and such might have them look outside, projects which do that should be fixed to not). That is a change to packaging does not need a rebuild at all (or at least that’s the ideal).

5

Thanks Ben also for your explanations!

Indeed I finished now the entire superbuild - only to be back at the same problem again. But I see now that fixing the problem in fixup_bundle_windows.py would solve the problem directly - without a rerun of the superbuild.

In order to track down the problem this is the print that I inserted into that py file:

        for dep in self._get_dependencies():
            if msvc_runtimes.match(dep):
                continue
            
            print '***CHECKING %s' % (dep)

            deplib = Library.from_reference(dep, self)
            if deplib is not None:
                collection[dep] = deplib

The resulting error message is now the following (extract):

1: CPack: Create package using NSIS
1: CPack Verbose: Read description file: C:/Program Files/CMake/share/cmake-3.11/Templates/CPack.GenericDescription.txt
1: CPack Verbose: [NSIS] requested component grouping = ONE_PER_GROUP
1: CPack Verbose: Remove toplevel directory: C:/dev/atgsb/blend/_CPack_Packages/win64/NSIS
1: CPack: Install projects
1: CPack: - Install project: cpack-athosgeo
1: CPack Verbose: Install configuration: “Release”
1: ***CHECKING vtkPVServerManagerApplication-pv5.6.dll
1: ***CHECKING vtkUtilitiesPythonInitializer-pv5.6.dll
1: ***CHECKING vtkPVServerImplementationCore-pv5.6.dll
1: ***CHECKING vtkPVClientServerCoreCore-pv5.6.dll
1: ***CHECKING vtkParallelCore-pv5.6.dll
1: ***CHECKING vtkCommonCore-pv5.6.dll
1: ***CHECKING VCRUNTIME140.dll
1: ***CHECKING api-ms-win-crt-runtime-l1-1-0.dll
1: CMake Error at C:/dev/atgsb/blend/cpack/athosgeo/NSIS/build/cmake_install.cmake:46 (message):
1: Failed to install pvdataserver:
1:
1: Traceback (most recent call last):
1:
1: File “C:/dev/atgsb/pvsb/superbuild/cmake/scripts/fixup_bundle.windows.py”, line 437, in
1: main(sys.argv[1:])
1: File “C:/dev/atgsb/pvsb/superbuild/cmake/scripts/fixup_bundle.windows.py”, line 429, in main
1: _install_binary(main_exe, is_excluded, bundle_dest, libdir, installed, manifest, dry_run=opts.dry_run)
1: File “C:/dev/atgsb/pvsb/superbuild/cmake/scripts/fixup_bundle.windows.py”, line 329, in _install_binary
1: deps = binary.dependencies.values()
1: File “C:/dev/atgsb/pvsb/superbuild/cmake/scripts/fixup_bundle.windows.py”, line 152, in dependencies
1: deplib = Library.from_reference(dep, self)
1: File “C:/dev/atgsb/pvsb/superbuild/cmake/scripts/fixup_bundle.windows.py”, line 199, in from_reference
1: raise RuntimeError(‘Unable to find the %s library from %s: %s’ % (ref, loader.path, ', '.join(paths)))
1:
1: RuntimeError: Unable to find the api-ms-win-crt-runtime-l1-1-0.dll library
1: from C:\dev\atgsb\blend\install\bin\pvdataserver.exe:
1: C:\dev\atgsb\blend\install\bin, lib,
1: C:/Qt2015/5.9.1/msvc2015_64/lib/cmake/Qt5/…/…/…/bin, C:\Program Files
1: (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64,
1: C:\Windows\Microsoft.NET\Framework64\v4.0.30319, C:\Program Files

In other words: shouldn’t there be a filter that directly filters out these files, like the two others that are already there? I mean something like:

        ms_pseudo_runtimes = re.compile('api-ms-win-*.dll')
        for dep in self._get_dependencies():
            if ms_pseudo_runtimes.match(dep):
                continue

(or the same integrated in the loop for the msvc_runtimes…)

Any good reason not to do this?

6

I actually tried now the following code variant:

        msvc_runtimes = re.compile('MSVC[A-Z][0-9]*\.dll')
        ms_pseudo_runtimes = re.compile('api-ms-win-.*\.dll')
        for dep in self._get_dependencies():
            if msvc_runtimes.match(dep):
                continue
            if ms_pseudo_runtimes.match(dep):
                continue

            deplib = Library.from_reference(dep, self)
            if deplib is not None:
                collection[dep] = deplib

…and the packaging went through successfully!

Still have to see how it behaves if I install in on another computer…