Paradoxon:导入时 Python 的 ctypes.CDLL 无声崩溃,但直接 运行 时不会崩溃 - 这怎么可能?

Paradoxon: silent crash on Python's ctypes.CDLL when importing, but not when running directly - how is this possible?

所以,作为一个 Linux 的人,我在 Windows 上遇到了一些我无法解释的非常令人费解的事情。

我有一个类似于此示例的项目结构:

D:\PROJECT
|
|   tolkien.py
|   __init__.py
|   
\---MiddleEarth
    |   gondor.py
    |   isengrad.c
    |   __init__.py
    |   
    \---lib
            isengrad.so

问题: 我将isengrad.c编译成共享库isengrad.so,然后在gondor.py中加载它。我的目标是将 gondor.py 导入 tolkien.py.
虽然 gondor.py 运行 直接 运行 时完美无缺,但当我导入它时,代码在我通过 ctypes.CDLL 加载共享库时退出,没有任何错误消息。

复制: 文件的内容(添加了一些 "status messages" 以跟踪问题发生的位置):

isengrad.c:

int isengrad(int hobbit){
    return hobbit/2;
}

然后用

编译成 isengrad.so
D:\project>chdir MiddleEarth
D:\project\MiddleEarth>gcc -fPIC -shared -o lib/isengrad.so isengrad.c

然后在gondor.py:

中访问共享库
print("started gondor")

import os, ctypes
path_to_isengrad = "D:/project/MiddleEarth/lib/isengrad.so"  

print("gondor loads isengrad")
gondor = ctypes.CDLL(path_to_isengrad)     # <--- crashes here when imported, not when ran directly
print("gondor loaded isengrad")


gondor.isengrad.argtypes = (ctypes.c_int,)

def faramir(hobbit):
    catched_hobbits = gondor.isengrad(hobbit)
    return catched_hobbits

if __name__ == '__main__':
    print(faramir(5))
    print("gondor ran")

print("gondor finished")

然后导入 tolkien.py:

print("started tolkien")
from MiddleEarth import gondor
print("tolkien imported gondor")

got = gondor.faramir(4)
print(got)

print("tolkien worked")

现在检查当我直接使用 gondor.py VS 在 tolkien.py 中导入它时会发生什么:

D:\project>python MiddleEarth/gondor.py
started gondor
gondor loads isengrad
gondor loaded isengrad
2
gondor ran
gondor finished

D:\project>python tolkien.py
started tolkien
started gondor
gondor loads isengrad

D:\project>

直接运行宁它根本没有问题。但是导入它会导致整个事情在加载共享库时崩溃而没有任何文字和回溯。这怎么会发生?我什至硬编码了共享库的路径,所以不同的工作目录应该不是问题......我在 Kubuntu 上的同一个项目没有任何问题,所以这可能是一些 Windows -相关的东西。

环境:

从看到这个问题的那一刻起,我就想说这是未定义的行为 (UB). Python 附带其 C 运行时 (UCRTLib),而 Cygwin .dll 自带。在进程中混合编译器和 C 运行时,通常是灾难的根源。
我找到了一个官方说法[Cygwin]: 6.15. Can I link with both MSVCRT*.DLL and cygwin1.dll?强调是我的):

No, you must use one or the other, they are mutually exclusive.

查看 [SO]: How to circumvent Windows Universal CRT headers dependency on vcruntime.h (@CristiFati's answer) 了解有关 MSVCRT*.DLL

的更多详细信息

现在,UB 的美妙之处在于它描述了一种看似随机的行为。

我已经准备了一个综合示例(稍微修改了您的代码)。

isengrad.c:

#if defined(_WIN32)
#  define ISENGRAD_EXPORT_API __declspec(dllexport)
#else
#  define ISENGRAD_EXPORT_API
#endif


ISENGRAD_EXPORT_API int isengrad(int hobbit) {
    return hobbit / 2;
}

script0.py:

#!/usr/bin/env python3

import sys
import ctypes


dll_name = "./lib/isengrad_{0:s}_{1:03d}.dll".format(sys.argv[1][:3] if sys.argv else sys.platform[:3].lower(), ctypes.sizeof(ctypes.c_void_p) * 8)
print("Attempting to load: {0:s}".format(dll_name))
isengrad_dll = ctypes.CDLL(dll_name)
print("DLL Loaded")


def main():
    isengrad_func = isengrad_dll.isengrad
    isengrad_func.argtypes = [ctypes.c_int]
    isengrad_func.restype = ctypes.c_int

    res = isengrad_func(46)
    print("{0:s} returned {1:}".format(isengrad_func.__name__, res))


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main()
    print("\nDone.")

script1.py:

#!/usr/bin/env python3

import sys
import script0


def main():
    pass


if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main()
    print("\nDone.")

输出:

  • 我将使用 3 windows:
    • cmd - Win (32bit and 64bit)
    • CygwinMintty
      • 64位
      • 32位
  • 请注意,即使我将每个内容粘贴到一个块中(以避免分散它们),我也会在 运行 命令
  • 时在它们之间切换
  • Cygwin 32位:

    [cfati@cfati-5510-0:/cygdrive/e/Work/Dev/Whosebug/q056855348]> ~/sopr.sh
    *** Set shorter prompt to better fit when pasted in Whosebug (or other) pages ***
    
    [032bit prompt]> gcc -shared -fPIC -o lib/isengrad_cyg_032.dll isengrad.c
    [032bit prompt]>  ls lib/*.dll
    lib/isengrad_cyg_032.dll  lib/isengrad_cyg_064.dll  lib/isengrad_win_032.dll  lib/isengrad_win_064.dll
    [032bit prompt]>
    [032bit prompt]> python3 script0.py cyg
    Attempting to load: ./lib/isengrad_cyg_032.dll
    DLL Loaded
    Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin
    
    isengrad returned 23
    
    Done.
    [032bit prompt]>
    [032bit prompt]> python3 script1.py cyg
    Attempting to load: ./lib/isengrad_cyg_032.dll
    DLL Loaded
    Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin
    
    
    Done.
    [032bit prompt]>
    [032bit prompt]> python3 script0.py win
    Attempting to load: ./lib/isengrad_win_032.dll
    DLL Loaded
    Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin
    
    isengrad returned 23
    
    Done.
    [032bit prompt]>
    [032bit prompt]> python3 script1.py win
    Attempting to load: ./lib/isengrad_win_032.dll
    DLL Loaded
    Python 3.6.4 (default, Jan  7 2018, 17:45:56) [GCC 6.4.0] 32bit on cygwin
    
    
    Done.
    
  • Cygwin 64位:

    [cfati@cfati-5510-0:/cygdrive/e/Work/Dev/Whosebug/q056855348]> ~/sopr.sh
    *** Set shorter prompt to better fit when pasted in Whosebug (or other) pages ***
    
    [064bit prompt]>  gcc -shared -fPIC -o lib/isengrad_cyg_064.dll isengrad.c
    [064bit prompt]> ls lib/*.dll
    lib/isengrad_cyg_032.dll  lib/isengrad_cyg_064.dll  lib/isengrad_win_032.dll  lib/isengrad_win_064.dll
    [064bit prompt]>
    [064bit prompt]> python3 script0.py cyg
    Attempting to load: ./lib/isengrad_cyg_064.dll
    DLL Loaded
    Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin
    
    isengrad returned 23
    
    Done.
    [064bit prompt]>
    [064bit prompt]> python3 script1.py cyg
    Attempting to load: ./lib/isengrad_cyg_064.dll
    DLL Loaded
    Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin
    
    
    Done.
    [064bit prompt]>
    [064bit prompt]> python3 script0.py win
    Attempting to load: ./lib/isengrad_win_064.dll
    DLL Loaded
    Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin
    
    isengrad returned 23
    
    Done.
    [064bit prompt]>
    [064bit prompt]> python3 script1.py win
    Attempting to load: ./lib/isengrad_win_064.dll
    DLL Loaded
    Python 3.6.8 (default, Feb 14 2019, 22:09:48) [GCC 7.4.0] 64bit on cygwin
    
    
    Done.
    
  • cmd:

    [cfati@CFATI-5510-0:e:\Work\Dev\Whosebug\q056855348]> sopr.bat
    *** Set shorter prompt to better fit when pasted in Whosebug (or other) pages ***
    
    [prompt]> dir /b lib
    
    [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community17\VC\Auxiliary\Build\vcvarsall.bat" x64
    **********************************************************************
    ** Visual Studio 2017 Developer Command Prompt v15.9.14
    ** Copyright (c) 2017 Microsoft Corporation
    **********************************************************************
    [vcvarsall.bat] Environment initialized for: 'x64'
    
    [prompt]> cl /nologo /DDLL isengrad.c  /link /NOLOGO /DLL /OUT:lib\isengrad_win_064.dll
    isengrad.c
       Creating library lib\isengrad_win_064.lib and object lib\isengrad_win_064.exp
    
    [prompt]>
    [prompt]> "c:\Install\x86\Microsoft\Visual Studio Community17\VC\Auxiliary\Build\vcvarsall.bat" x86
    **********************************************************************
    ** Visual Studio 2017 Developer Command Prompt v15.9.14
    ** Copyright (c) 2017 Microsoft Corporation
    **********************************************************************
    [vcvarsall.bat] Environment initialized for: 'x86'
    
    [prompt]> cl /nologo /DDLL isengrad.c  /link /NOLOGO /DLL /OUT:lib\isengrad_win_032.dll
    isengrad.c
       Creating library lib\isengrad_win_032.lib and object lib\isengrad_win_032.exp
    
    [prompt]> dir /b lib\*.dll
    isengrad_cyg_032.dll
    isengrad_cyg_064.dll
    isengrad_win_032.dll
    isengrad_win_064.dll
    
    [prompt]> set _PATH=%PATH%
    
    [prompt]> :: Python 32bit
    [prompt]> set PATH=%_PATH%;e:\Install\x86\Cygwin\Cygwin\Version\bin
    
    [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py win
    Attempting to load: ./lib/isengrad_win_032.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32
    
    isengrad returned 23
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py win
    Attempting to load: ./lib/isengrad_win_032.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32
    
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script0.py cyg
    Attempting to load: ./lib/isengrad_cyg_032.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32
    
    isengrad returned 23
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_032_03.07.03_test0\Scripts\python.exe" script1.py cyg
    Attempting to load: ./lib/isengrad_cyg_032.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 21:26:53) [MSC v.1916 32 bit (Intel)] 32bit on win32
    
    
    Done.
    
    [prompt]> :: Python 64bit
    [prompt]> set PATH=%_PATH%;c:\Install\x64\Cygwin\Cygwin\AllVers\bin
    
    [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py win
    Attempting to load: ./lib/isengrad_win_064.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32
    
    isengrad returned 23
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py win
    Attempting to load: ./lib/isengrad_win_064.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32
    
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script0.py cyg
    Attempting to load: ./lib/isengrad_cyg_064.dll
    DLL Loaded
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32
    
    isengrad returned 23
    
    Done.
    
    [prompt]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" script1.py cyg
    Attempting to load: ./lib/isengrad_cyg_064.dll
    
    [prompt]>
    [prompt]> echo %errorlevel%
    -1073741819
    

正如所见,交叉编译器 .exe - .dll 在 7 个(共 8 个)案例中工作(在 64位 Win Pythonscript1.py ), 而同一个编译器在所有 8 个中都工作。

因此,我建议在使用此类环境时,尽量使用于构建各个部分的编译器保持一致一致(或兼容 至少)。



更新#0

我只是想到了 64 位 上可能出错的原因:sizeof(long) 通常不同(以下大小以字节为单位):

  • 4Win
  • 8Cygwin 上(在 Nix, 一般)

sizeof(long double)(即 2 * sizeof(long))也一样。

因此,如果 Cygwin .dll 公开了一些 long 值大于 2 ** 64 (1 << 64), 它将在Win进程中被截断,在这种情况下可能会发生崩溃。从理论上讲,这种情况应该也会影响相反的情况,但事实并非如此。

还有其他因素可能导致此行为,如默认内存对齐等。