从 Travis CI（和其他 CI 工具）的源代码中缓存 git 与 Composer 安装的存储库？

Question

我在 GitHub 上托管的几个 Symfony 包上工作，并使用 Travis CI 自动测试。

测试中最长的部分是 Composer 安装要求。

我configured Travis CI to install packages with composer update --prefer-dist and cache the $HOME/.composer/cache directory. Tests took a total time of 18 minutes，感谢缓存：

Installing doctrine/lexer (v1.0.1)

Loading from cache

但是一周前我看到了来自 Composer 的消息：

Installing doctrine/lexer (v1.0.1)

Downloading: Connecting... Failed to download doctrine/lexer from dist: Could not authenticate against github.com

我changed the configuration to composer update --prefer-source because of this. This seems to be a common practice across Symfony bundles. The tests suite took 28 minutes.

我知道我可以在 Travis CI 中注册 GitHub 键，以避免在使用 Composer --prefer-dist 选项时出现 API 限制。

它们是否有其他缓存依赖项的方法？例如，通过在缓存中克隆依赖项存储库？

Answer 1

GitHub 已删除 API 速率限制，composer 现在可以与 --prefer-dist 一起使用，并且可以缓存 zip 文件。这是 .travis.yml 中的配置示例：

cache:
  directories:
    - $HOME/.composer/cache

# …

install:
  - composer update --prefer-dist

公告如下：

Hi Niels and Jordi,

We've spent some time digging in, considering all the options to resolve your issues, weighing your needs against infrastructure strain and availability.

I'm happy to report that our Infrastructure team believes that due to their work on our Git backend since these APIs were introduced, we're now able to drop these rate limits on the API side. We deployed those changes a couple of hours ago. Getting an archive link 1 via the API will no longer count against your hourly rate limit (authenticated or not). This should make Composer installs happy.

Let us know if you see any funny business.

Cheers,

Wynn Netherland
Platform Engineering Manager, GitHub

Source.

Answer 2

我测试了 vendor/ 目录的缓存并且它有效。我使用 tar 来创建未压缩的存档 $HOME/vendor-cache/ 并为此目录配置了 Travis CI。

命令有两个目标：

如果可用，从缓存中提取 vendor/
测试后将vendor/放入缓存

这里是一个例子 .travis.yml 文件：

sudo: false

language: php

cache:
  directories:
    - $HOME/.composer/cache
    # This is where vendor/ backups will be stored
    - $HOME/vendor-cache/

php:
  - […]

env:
  - SYMFONY_VERSION="2.7.*"
  - SYMFONY_VERSION="2.8.*"
  - SYMFONY_VERSION="3.0.*"

before_install:
 # Create an hash corresponding to the PHP version and the dependencies
 - tohash="${SYMFONY_VERSION}"
 - cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
 # Extract cache archive if the file exists
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then tar -xf $HOME/vendor-cache/$cachefile ; fi

install:
  - composer self-update
  - composer update --profile --no-progress

script: php ./vendor/bin/phpunit

# Create cache archive from vendor/ directory
before_cache:
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; rm -fv $HOME/vendor-cache/$cachefile ; fi
 - tar -cf $HOME/vendor-cache/$cachefile vendor/

这是一个带有更详细输出的完整注释 .travis.yml 文件：

sudo: false

language: php

cache:
  directories:
    - $HOME/.composer/cache
    # This is where vendor/ backups will be stored
    - $HOME/vendor-cache/

git:
  depth: 5

php:
  - […]

env:
  - SYMFONY_VERSION="2.7.*"
  - SYMFONY_VERSION="2.8.*"
  - SYMFONY_VERSION="3.0.*"

before_install:
 # Create an hash corresponding to the PHP version and the dependencies
 - echo "Vendor cache content:" ; ls -lh $HOME/vendor-cache/
 - echo "Values used for hash:"
 - tohash="${SYMFONY_VERSION}"
 - echo "$tohash"
 - cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
 - echo "cachefile = ${cachefile}"
 # Extract cache archive if the file exists
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Extract cache archive"; tar -xf $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "Cache archive does not exist" ; fi
 - if [[ -d vendor/ ]]; then echo "Size of vendor directory extracted from cache:" ; du -hs vendor/; else echo "vendor/ directory does not exist"; fi

install:
  - composer self-update
  - composer update --profile --no-progress

script: php ./vendor/bin/phpunit

# Create cache archive from vendor/ directory
before_cache:
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Delete previous cache archive"; rm -fv $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "No cache archive to delete" ; fi
 - echo "Create cache archive" ; tar -cf $HOME/vendor-cache/$cachefile vendor/ ; echo "Done"
 - echo "Size of cache archive:" ; ls -lh $HOME/vendor-cache/$cachefile

通过使用此方法，composer update 在没有缓存的情况下获得了 30 seconds, instead of about 2 minutes（请注意，比较并不完美，应用了一些其他更改，但这仍然是一个不错的估计）。

最好在您第一次启动构建时限制并行构建的数量，这样缓存就不会出现竞争条件。

从 Travis CI（和其他 CI 工具）的源代码中缓存 git 与 Composer 安装的存储库？

Cache git repositories installed with Composer from source on Travis CI (and other CI tools)?

php

github

travis-ci

composer-php