Failing to run compiled Python extension

I am trying to build a combined Python/C++ project on the Windows CI server. This uses PyBind to build an extension module (a DLL) which should be importable from Python. I’ve already gotten it working on Linux and Mac (although currently have those builds commented out to save time while testing on Windows).

When I try to build it in the Travis Windows environment I’m able to compile and link the extension just fine. However, When I try testing the C++ routines from within C++, that also works. However, when I try importing the compiled Python extension from within Python, it fails to find it. This is despite my having confirmed that the module is located within the Python path. In fact, as an experiment, I created a trivial pure Python module in the same directory as the compiled one and was able to import the former just fine. It seems that Python is failing to recognise the compiled extension module.

I use Linux for my day-to-day work, but I borrowed a colleague’s laptop which was running Windows. I used as similar a build environment to the Windows CI server as possible, with Git-bash and mingw. The main difference was that Python had been installed via Conda rather than Chocolatey. On this system everything compiled and ran seamlessly, leaving me at something of a loss.

You can find my configurations in my Travis config file and the script which it calls. The build itself is defined within the CMakeList.txt files in that repository.

tests\py\testsuite.py:3: in <module>
    from .numericaltests import NumericalDataUnitTest
tests\py\numericaltests.py:4: in <module>
    import peakingduck as pkd
peakingduck\__init__.py:1: in <module>
    import peakingduck.util as util
peakingduck\util\__init__.py:2: in <module>
    from PEAKINGDUCK.util import *
E   ImportError: DLL load failed while importing PEAKINGDUCK: The specified module could not be found.

Is the last name intended to be in CAPS?


Other than that, “The specified module could not be found.” may mean that a dependency DLL is not found.

Yes.

I take it you mean that PEAKINGDUCK.pyd is trying to load some DLL which it can’t find? To diagnose that I’ve tried running depends and dumpbin on the CI, but in both cases I got the complaint that these commands could not be found.

https://travis-ci.com/thomasms/peakingduck/jobs/294999409
https://travis-ci.com/thomasms/peakingduck/jobs/295020124

If it is of any value, when I run the corresponding command on my Linux box, the only dependencies should be libm, libc, and compiler libraries. Something is very wrong indeed if those can’t be found on the CI.

dumpbin is not on PATH, use info on the provided links or find /c to locate it. Others are not present, you need to download them yourself. I deliberately didn’t mention depends among the links because it’s obsolete (doesn’t support API sets), use Dependencies instead.

Whether something is “wrong” or not, only one way to find out.

Downloading and running dependencies printed out an incredibly complicated hierarchy (see https://travis-ci.com/thomasms/peakingduck/jobs/295345741#L339). There are a large number of missing modules:

  • ext-ms-win-shell32-shellcom-l1-1-0.dll
  • api-ms-win-core-comm-l1-1-0.dll
  • api-ms-win-core-string-obsolete-l1-1-0.dll
  • ext-ms-win-security-chambers-l1-1-0.dll
  • api-ms-win-core-string-l2-1-0.dll
  • api-ms-win-core-stringansi-l1-1-0.dll
  • api-ms-win-core-version-l1-1-0.dll
  • ext-ms-win-rometadata-dispenser-l1-1-0.dll
  • CoreUIComponents.dll
  • TextInputFramework.dll
  • ext-ms-win-appmodel-viewscalefactor-l1-1-0.dll
  • ext-ms-win-ntuser-windowclass-l1-1-0.dll
  • api-ms-win-core-version-private-l1-1-0.dll
  • RMCLIENT.dll
  • ext-ms-win-com-psmregister-l1-1-0.dll
  • ext-ms-win-com-suspendresiliency-l1-1-0.dll
  • ext-ms-win-appmodel-deployment-l1-1-0.dll
  • ext-ms-win-appmodel-usercontext-l1-1-0.dll
  • ext-ms-win-ui-viewmanagement-l1-1-0.dll
  • CoreUIComponents.dll
  • ext-ms-win-dwmapidxgi-ext-l1-1-0.dll
  • ext-ms-mf-pal-l2-1-0.dll
  • dcomp.dll
  • D3DSCache.dll
  • ext-ms-win-wer-xbox-l1-1-0.dll
  • ext-ms-win-smbshare-browserclient-l1-1-0.dll
  • api-ms-win-security-sddlparsecond-l1-1-0.dll
  • ext-ms-win-core-winrt-remote-l1-1-0.dll
  • wpaxholder.dll
  • policymanager.dll
  • ext-ms-win-shell32-shellfolders-l1-1-0.dll
  • ext-ms-win-shell-knownfolderext-l1-1-0.dll
  • efswrt.dll
  • ext-ms-win-shell-tabbedtitlebar-l1-1-0.dl
  • ext-ms-onecore-appmodel-emclient-l1-1-0.dll
  • ext-ms-win-rtcore-minuser-private-ext-l1-1-0.dll
  • windows.globalization.fontgroups.dll
  • FVEAPI.dll
  • ContactActivation.dll
  • cryptngc.dll
  • edpauditapi.dll
  • dsreg.dll
  • edputil.dll
  • SystemEventsBrokerClient.dll
  • CLDAPI.dll
  • ext-ms-onecore-defaultdiscovery-l1-1-0.dl
  • ext-ms-win-audiocore-pal-l1-2-0.dll
  • DEVMGR.DLL
  • SHDOCVW.dll
  • EFSADU.dll
  • elscore.dll
  • ext-ms-win-shell-shlwapi-l1-1-0.dll
  • ext-ms-windowscore-deviceinfo-l1-1-0.dll
  • NETPLWIZ.dll

There are almost certainly others which I haven’t noted down. These all seem to be neede by USER32.dll and msvcrt.dll, or one of those two libraries’ own dependencies.

I have created a minimal-reproducer for this: https://github.com/cmacmackin/travis-windows-pybind.

See the build-log here: https://travis-ci.com/cmacmackin/travis-windows-pybind/jobs/295407821

1 Like

Bottom line: change compiler flags to link to libgcc and libstdc++ statically.
Or place them (with any transitive MinGW-specific dependencies) alongside the .pyd and distribute with your module.


I’ve diagnosed the problem with Process Monitor using the following code.
This is the most reliable way because you get to see what the system is actually searching for, and where, rather than look at metadata and make assumptions. Since Windows’ DLL search process is compilcated and very settings-dependent, the latter proved unreliable.

# FTP_* are secret variables for an FTP server that I run on my machine when needed
- curl -f -O ftp://$FTP_USER:$FTP_PASSWD@$FTP_SERVER/procmon.exe
- ./procmon //AcceptEula //Quiet //Minimized //BackingFile error.pml & (until test -f *.pml; do sleep 1; done)
- <command under test>
- ./procmon //Terminate
- gzip -v *.pml
- curl -T "$(perl -e 'print "{".join(",",@ARGV)."}"' *.gz)" ftp://$FTP_USER:$FTP_PASSWD@$FTP_SERVER/

The relevant part of the resulting .pml I got onto my machine for examination is:

"Time of Day","Process Name","PID","TID","Operation","Path","Result","Detail"
"22:46:05,0651278","python.exe","4852","3816","Load Image","C:\Users\travis\build\native-api\travis-windows-pybind\build\CPPMATH.cp38-win_amd64.pyd","SUCCESS","Image Base: 0x68cc0000, Image Size: 0x3d000"
"22:46:05,0652375","python.exe","4852","3816","CloseFile","C:\Users\travis\build\native-api\travis-windows-pybind\build\CPPMATH.cp38-win_amd64.pyd","SUCCESS",""
"22:46:05,0653272","python.exe","4852","3816","QueryOpen","C:\Users\travis\build\native-api\travis-windows-pybind\build\libgcc_s_seh-1.dll","NAME NOT FOUND",""
"22:46:05,0654100","python.exe","4852","3816","QueryOpen","C:\Python38\libgcc_s_seh-1.dll","NAME NOT FOUND",""
"22:46:05,0655351","python.exe","4852","3816","QueryOpen","C:\Windows\System32\libgcc_s_seh-1.dll","NAME NOT FOUND",""

So, the module it fails to find is libgcc_s_seh-1.dll, and for some reason, it only searches for it in a few locations rather than everything on PATHwhich does include C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin where this DLL is located.

Looking at the stacktrace of the events in Process Monitor shows that the .pyd is being loaded from python38.dll with LoadLibraryExW. Searching Python codebase (tag 3.8.2 as this is the version you are installing) for LoadLibraryEx and then looking for anything related to module loading in the results finds this peculiar code:

        /* bpo-36085: We use LoadLibraryEx with restricted search paths
           to avoid DLL preloading attacks and enable use of the
           AddDllDirectory function. We add SEARCH_DLL_LOAD_DIR to
           ensure DLLs adjacent to the PYD are preferred. */
        Py_BEGIN_ALLOW_THREADS
        hDLL = LoadLibraryExW(wpathname, NULL,
                              LOAD_LIBRARY_SEARCH_DEFAULT_DIRS |
                              LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);

which explains the restricted search path.

Since those libraries are not stocked with Windows or CPython but are rather specific to the MinGW toolchain, you need to link to them statically (or place, together with their dependencies, alongside the .pyd, a location which is searched).

The restricted search path is actually proving very useful in that you can detect such dependencies early!

1 Like

Thanks for the detailed investigation and explanation. One more question which I’m curious about. As I said, I was able to successfully build and run this code on a colleague’s Windows laptop. Do you have any idea why it was able to work there? I guess the default search paths were set up differently?

Can’t say for sure without seeing their configuration. My best guess is because this logic is new in 3.8.0.

Trying to statically link the GCC libraries didn’t work, unfortunately. It seemed to cause a segfault or something similarly low-level.

https://travis-ci.com/cmacmackin/travis-windows-pybind/builds/152421188#L176

Python crashed without even giving it’s usual stack trace.

The command "$PY_CMD -m pytest ./test_bindings.py" exited with 127.
Done. Your build exited with 1.

Windows fatal exception: code 0xc0000374


Current thread 0x00001214 (most recent call first):

  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1101 in create_module
  File "<frozen importlib._bootstrap>", line 556 in module_from_spec
  File "<frozen importlib._bootstrap>", line 657 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\travis\build\cmacmackin\travis-windows-pybind\test_bindings.py", line 3 in <module>
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\assertion\rewrite.py", line 143 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\py\_path\local.py", line 701 in pyimport
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\python.py", line 493 in _importtestmodule
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\python.py", line 425 in _getobj
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\python.py", line 249 in obj
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\python.py", line 441 in _inject_setup_module_fixture
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\python.py", line 428 in collect
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\runner.py", line 257 in <lambda>
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\runner.py", line 237 in from_call
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\runner.py", line 257 in pytest_make_collect_report
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\runner.py", line 379 in collect_one_node
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 721 in genitems
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 496 in _perform_collect
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 458 in perform_collect
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 256 in pytest_collection
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 246 in _main
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 197 in wrap_session
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\main.py", line 240 in pytest_cmdline_main
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\callers.py", line 187 in _multicall
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 84 in <lambda>
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\manager.py", line 93 in _hookexec
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pluggy\hooks.py", line 286 in __call__
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\_pytest\config\__init__.py", line 92 in main
  File "C:\Users\travis\AppData\Roaming\Python\Python38\site-packages\pytest\__main__.py", line 7 in <module>
  File "C:\Python38\lib\runpy.py", line 86 in _run_code
  File "C:\Python38\lib\runpy.py", line 193 in _run_module_as_main

There’s a compiler warning. Since compiler warnings aren’t just for show, I would look into that.

FWIW, 0xC0000374 is STATUS_HEAP_CORRUPTION.

Also “MSYS Makefiles” strikes me as odd (you can’t (reliably) invoke other Cygwin forks from Git Bash directly). You probably want “MinGW makefiles”.
And the fact that you override parts of the toolchain – whether it works or not, it’s clearly an unsupported setup.

FWIW, there’s now official guidance on how to invoke commands in MSYS and MinGW environment. I don’t know if CMake makes that unnecesary by setting up the correct environment for you or not.

Imprint