在 python 包的 dependency_links 中定义镜像存储库

Question

Python 的包管理器允许通过安装脚本的 dependency_links 参数定义非 PyPI 依赖项。但是也可以用类似的方式定义 mirrors 吗？（所以，相同的包，相同的版本，一切都相同，-只是在两个不同的存储库中，-我希望第二个作为后备，以防第一个失败 - 例如主机无法访问，身份验证失败等）。

UPD

我确实已经尝试为 dependency_links 中的同一个包定义多个 URL。不幸的是，这并不像我希望的那样有效。如果第一个被解析为与所请求的 package/version 正确匹配的存储库由于某种原因失败，那么整个安装脚本就会失败（即它不会尝试遍历 所有其他 正确匹配，直到找到一个好的匹配，或者所有匹配都失败）。

Answer 1

查看 setuptools' 源代码，默认代码仅对 dependency_links 中的 URL 进行语法验证，但实际文件下载尝试时发生的所有错误都会停止脚本执行.在下面的示例中，默认包索引 impl 被覆盖，因此除了 URL 语法验证之外，还会尝试下载目标文件。如果失败，则流程切换到下一个 URL.

from distutils.errors import DistutilsError
import os
import tempfile

from setuptools import setup
from setuptools.command.easy_install import easy_install
from setuptools.package_index import PackageIndex as PackageIndexOrig
import faker


class PackageIndex(PackageIndexOrig):

    def url_ok(self, url, fatal=False):
        if super().url_ok(url, fatal):
            try:
                tmpfile = os.path.join(tempfile.mkdtemp(prefix='url-ok-check-'), 'file.out')
                self._attempt_download(url, tmpfile)
                return True
            except Exception as ex:
                msg = 'Download error for %s: %s' % (url, ex)
                if fatal:
                    raise DistutilsError(msg)
                self.warn(msg)
        return False


f = faker.Faker()
fake_urls = [f.url() + '#egg=django' for i in range(10)]
urls = [fake_urls + ['https://github.com/django/django/archive/stable/2.0.x.zip#egg=django']

easy_install.create_index = PackageIndex

setup(
    name='spam',
    packages=['spam'],
    install_requires=['django'],
    dependency_links=urls,
)

测试一下：

$ pip install faker
$ python setup.py install --verbose
running install
...
Installed /Users/hoefling/.virtualenvs/Whosebug/lib/python3.6/site-packages/spam-0.0.0-py3.6.egg
Downloading https://harris.com#egg=django
Download error for https://harris.com#egg=django: Unexpected HTML page found at https://harris.com#egg=django
Downloading https://www.torres.com#egg=django
Download error for https://www.torres.com#egg=django: Download error for https://www.torres.com#egg=django: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:777)
Downloading https://smith.com/#egg=django
Download error for https://smith.com/#egg=django: Download error for https://smith.com/#egg=django: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
Downloading http://tucker.info/#egg=django
Download error for http://tucker.info/#egg=django: Unexpected HTML page found at http://tucker.info/#egg=django
Downloading https://christian-murphy.org/#egg=django
Download error for https://christian-murphy.org/#egg=django: Download error for https://christian-murphy.org/#egg=django: [Errno 8] nodename nor servname provided, or not known
Downloading http://www.ramirez.com/#egg=django
Download error for http://www.ramirez.com/#egg=django: Unexpected HTML page found at http://www.ramirez.com/#egg=django
Downloading http://www.perez-davis.com/#egg=django
Download error for http://www.perez-davis.com/#egg=django: Download error for http://www.perez-davis.com/#egg=django: [Errno 8] nodename nor servname provided, or not known
Downloading https://ramirez.org/#egg=django
Download error for https://ramirez.org/#egg=django: Download error for https://ramirez.org/#egg=django: [Errno 8] nodename nor servname provided, or not known
Downloading https://www.bridges.com/#egg=django
Download error for https://www.bridges.com/#egg=django: Download error for https://www.bridges.com/#egg=django: [Errno 61] Connection refused
Downloading https://bryant.org/#egg=django
Download error for https://bryant.org/#egg=django: Download error for https://bryant.org/#egg=django: [Errno 61] Connection refused
Downloading http://porter-griffith.com/#egg=django
Download error for http://porter-griffith.com/#egg=django: Download error for http://porter-griffith.com/#egg=django: [Errno 8] nodename nor servname provided, or not known
Downloading https://www.hooper.net/#egg=django
Download error for https://www.hooper.net/#egg=django: Download error for https://www.hooper.net/#egg=django: [Errno 61] Connection refused
Downloading https://github.com/django/django/archive/stable/2.0.x.zip#egg=django
Processing dependencies for spam==0.0.0
Searching for django
Best match: django [unknown version]
Downloading https://github.com/django/django/archive/stable/2.0.x.zip#egg=django
Downloading https://github.com/django/django/archive/stable/2.0.x.zip#egg=django
Processing 2.0.x.zip
...

覆盖 PackageIndex 的缺点是，一旦找到有效的 link，脚本就会下载分发版两次 - 第一次是验证 link，然后是实际下载它安装。这可以通过编写更多 fine-grained 检查而不是仅使用 _attempt_download() 方法来避免。或者您可以存储对下载文件的引用，然后在 _download_url() 方法中重新使用它：

class PackageIndex(PackageIndexOrig):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._attempted = None

    def url_ok(self, url, fatal=False):
        if super().url_ok(url, fatal):
            try:
                tmpfile = os.path.join(tempfile.mkdtemp(prefix='url-ok-check-'), 'file.out')
                self._attempted = self._attempt_download(url, tmpfile)
                return True
            except Exception as ex:
                msg = 'Download error for %s: %s' % (url, ex)
                if fatal:
                    raise DistutilsError(msg)
                self.warn(msg)
        return False

    def _download_url(self, scheme, url, tmpdir):
        if self._attempted is not None:
            (result, self._attempted) = (self._attempted, None)
            return result
        return super()._download_url(scheme, url, tmpdir)

在 python 包的 dependency_links 中定义镜像存储库

Define mirror repositories in dependency_links of python package

python

pip

setuptools