在运行时构建多个轮子

Question

我想在运行时构建轮子，其中包含一些脚本和一些有效负载数据。例如。在此示例中，target 文件夹包含两个简单的构建 build_123 和 build_124 以打包为轮子。

main_project
├── __init__.py
├── whl_util.py               # wheel building script posted below
target/
├── build_123/                # contains one build to be packaged as a whl
│   └── mypkg
|       ├── __init__.py
│       ├── data
|       |   ├── __init__.py
|       |   └── mat.json
│       └── main
|           ├── __init__.py
|           └── dumpmat.py
└── build_124/                # contains another build to be packaged as a whl
    └── mypkg
        ├── ...

在我的场景中，这个轮子被用作输出格式，打包不是主要进展。 wheel 打包应该被视为一个简单的 IO 操作，它读取构建文件夹并输出一个 wheel，除此之外没有任何副作用。为了执行此任务，我想出了这个解决方案：

# main_project/whl_util.py
from setuptools import setup, find_packages
import sys
import shutil
import os

def bdist_wheel(build_dir=".", dist_dir=None):
    # backing up argv to restore them afterwards
    argv_bak = sys.argv[:]

    # clear args from running script with "bdist_wheel"
    file = sys.argv[0]
    sys.argv.clear()
    sys.argv.extend([file, "bdist_wheel"])

    if dist_dir is not None and "--dist-dir" not in sys.argv:
        sys.argv.extend(["--dist-dir", dist_dir])

    sys.argv.extend(["clean", "--all"])

    setup(
        name="mypkg",
        version=0.1,
        packages=find_packages(build_dir),
        install_requires=[],
        include_package_data=True,
        package_dir={'': build_dir},
        package_data={"mypkg.data": ["mat.json"]}
    )

    # restore args
    sys.argv.clear()
    sys.argv.extend(argv_bak)


def main():
    # Adding main method here for testing.
    # As mentioned in my actual scenario the wheels should be built as an output format at runtime
    print("BUILD 123")
    bdist_wheel("target/build_123", dist_dir="target/dist_123")
    print("BUILD 124")
    bdist_wheel("target/build_124", dist_dir="target/dist_124")

if __name__ == "__main__":
    main()

我也不太喜欢通过 sys.argv 将参数传递给 setuptools 的方式，但这似乎是唯一的方式。然而，主要问题是，第一个轮子正常构建，而 bdist_wheel/ setup 的第二次调用引发错误：

python3 -m main_project.whl_util
BUILD 123
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/mypkg
copying target/build_123/mypkg/__init__.py -> build/lib/mypkg
creating build/lib/mypkg/data
copying target/build_123/mypkg/data/__init__.py -> build/lib/mypkg/data
creating build/lib/mypkg/main
copying target/build_123/mypkg/main/__init__.py -> build/lib/mypkg/main
copying target/build_123/mypkg/main/dumpmat.py -> build/lib/mypkg/main
running egg_info
writing target/build_123/mypkg.egg-info/PKG-INFO
writing dependency_links to target/build_123/mypkg.egg-info/dependency_links.txt
writing top-level names to target/build_123/mypkg.egg-info/top_level.txt
writing manifest file 'target/build_123/mypkg.egg-info/SOURCES.txt'
copying target/build_123/mypkg/data/mat.json -> build/lib/mypkg/data
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/mypkg
copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
creating build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/mat.json -> build/bdist.linux-x86_64/wheel/mypkg/data
creating build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/dumpmat.py -> build/bdist.linux-x86_64/wheel/mypkg/main
running install_egg_info
Copying target/build_123/mypkg.egg-info to build/bdist.linux-x86_64/wheel/mypkg-0.1-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/mypkg-0.1.dist-info/WHEEL
creating 'target/dist_123/mypkg-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'mypkg/__init__.py'
adding 'mypkg/data/__init__.py'
adding 'mypkg/data/mat.json'
adding 'mypkg/main/__init__.py'
adding 'mypkg/main/dumpmat.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
BUILD 124
running bdist_wheel
running build
running build_py
copying target/build_124/mypkg/__init__.py -> build/lib/mypkg
error: could not create 'build/lib/mypkg/__init__.py': No such file or directory

Process finished with exit code 1

no such file or directory 错误表明 setuptools 模块会跟踪它已经创建的文件夹并假定这些文件夹仍然存在。然而，在构建第一个轮子后，clean 脚本将删除构建的文件夹（这是必要的，因为 setuptools 否则将重新使用该文件夹而不清除它）

我唯一可行的解决方案是在应用之前分叉进程 setup:

pid = os.fork()
if pid == 0:
    setup(...)
    sys.exit(0)
os.waitpid(pid, 0)

但是因为这看起来很脏而且我的主进程非常占用内存，所以我宁愿避免这种方法。

所以我的主要问题是： 有没有办法制造一个没有任何副作用的轮子？或者有没有办法在应用 setup 后重置 setuptools 模块的状态？在一个最佳世界中，我想在内存中创建轮子 PyFilesystem 并且只将轮子写入磁盘。

Answer 1

我不确定 setuptools 是否应该以这种方式使用。据我所知 pip 和合作。（wheel、setuptools 等）实际上没有 public API 或者至少没有友好的.

distlib 库看起来是一个很有前途的替代品，具有实际的 API。参见 distlib's documentation on "Using the wheel API"。

如果这不起作用，那么我可能会试一试：

subprocess.check_call([sys.executable, '-m', 'wheel', 'pack', 'target/build123'])

参见 wheel pack documentation。

subprocess.check_call([sys.executable, '-m', 'pip', 'wheel', 'target/build123'])

见pip wheel documentation. And the reasons why it can't be used with API calls are noted in the "Using pip from your program" section of pip's documentation

有一些类似的问题有一些有趣的想法：

How to run 'python setup.py install' from within Python?

也许在运行时动态生成 setup.py 可能会有所帮助。

在运行时构建多个轮子

Build multiple wheels at runtime

python

setuptools