将 Hashicorp Vault 与 GitLab 管道一起使用的最佳方法是什么？
What is the best way to use Hashicorp Vault with GitLab pipelines?
假设我想用 Vault 中的值创建一个变量。
variables:
$SSH_PRIVATE_KEY: `vault kv get -field=private_key project/production`
before_script:
- echo "$SSH_PRIVATE_KEY"
可以吗？
还有其他方法可以在管道内使用 Vault 机密吗？
2019 年 7 月的原始答案：
您可以看到它在 before/after 脚本步骤中使用，最后带有一个已撤销的令牌。
以 gitlab.eng.cleardata.com pub/pipelines/gcp-ci.yml 为例：
# Obtains credentials via vault (the gitlab-runner authenticates to vault using its AWS credentials)
# Configures the `gcloud` sdk and `kubectl` to authenticate to our *production* cluster
#
# Note: Do not override the before_script or the after_script in your job
#
.auth-prod: &auth-prod
image: cleardata/bionic
before_script:
- |
export CLUSTER_NAME=production
export CLUSTER_LOCATION=us-central1
export CLUSTER_PROJECT_ID=cleardata-production-cluster
- vault login -method=aws -path=gitlab-ci -no-print header_value=gitlab.eng.cleardata.com
- GCP_CREDS=$(vault read -field=private_key_data gitlab-ci/gcp/cleardata-production-cluster/key/deployment-key)
- gcloud auth activate-service-account --key-file=<(base64 -d <<<$GCP_CREDS)
- gcloud auth configure-docker
- gcloud beta container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_LOCATION --project $CLUSTER_PROJECT_ID
after_script:
- vault token revoke -self
2020 年 3 月更新：GitLab 12.9 支持此功能
HashiCorp Vault GitLab CI/CD Managed Application
GitLab wants to make it easy for users to have modern secrets management. We are now offering users the ability to install Vault within a Kubernetes cluster as part of the GitLab CI managed application process.
This will support the secure management of keys, tokens, and other secrets at the project level in a Helm chart installation.
参见 documentation and issue。
2020 年 4 月：GitLab 12.10：
Retrieve CI/CD secrets from HashiCorp Vault
In this release, GitLab adds support for lightweight JSON Web Token (JWT) authentication to integrate with your existing HashiCorp Vault.
Now, you can seamlessly provide secrets to CI/CD jobs by taking advantage of HashiCorp’s JWT authentication method rather than manually having to provide secrets as a variable in GitLab.
参见 documentation and issue。
参见 GitLab 13.4（2020 年 9 月）
仅限 Premium/Silver:
Use HashiCorp Vault secrets in CI jobs
In GitLab 12.10, GitLab introduced functionality for GitLab Runner to fetch and inject secrets into CI jobs. GitLab is now expanding the JWT Vault Authentication method by building a new secrets syntax in the .gitlab-ci.yml file. This makes it easier for you to configure and use HashiCorp Vault with GitLab.
https://about.gitlab.com/images/13_4/vault_ci.png -- Use HashiCorp Vault secrets in CI jobs
See Documentation and Issue.
参见 GitLab 13.9（2021 年 2 月）
Vault JWT (JSON Web Token) supports GitLab environments.
To simplify integrations with HashiCorp Vault, we’ve shipped
Vault JWT token support. From the launch, you could restrict access based on
data in the JWT. This release gives you a new dimension for restricting
access to credentials: the environment a job targets.
This release extends the existing Vault JWT token to support environment-based
restrictions too. As the environment name could be supplied by the user running
the pipeline, we recommend you use the new environment-based restrictions with the
already-existing ref_type values for maximum security.
See Documentation and Issue.
我们在构建器映像中嵌入了一个帮助脚本，它可以将 GitLab CI/CD 指向秘密的作业变量转换为包含 Vault 秘密的作业环境变量。在我们的例子中，我们还使用 appRole auth 方法来限制临时 Vault 访问令牌的有效性。
一个示例用例是：
I want a job env var "MY_SECRET_TOKEN" with a value from a Vault secret.
So I add a CI/CD variable called V_MY_SECRET_TOKEN="secret/<path>/<key>"
Then I insert a job step to retrieve the secret value and populate
the MY_SECRET_TOKEN with the value associated with the key.
在 GitLab 中添加到 CICD 作业设置的变量。
VAULT_ADDR=https://vault.example.com:8200
VAULT_ROLE_ID=db02de05-fa39-4855-059b-67221c5c2f63
VAULT_SECRET_ID=6a174c20-f6de-a53c-74d2-6018fcceff64
VAULT_VAR_FILE=/var/tmp/vault-vars.sh
添加到 .gitlab-ci.yml 作业定义的步骤。
script:
- get-vault-secrets-by-approle > ${VAULT_VAR_FILE}
- source ${VAULT_VAR_FILE} && rm ${VAULT_VAR_FILE}
这里引用了 get-vault-secrets-by-approle helper script we use.
Here is a writeup of the thinking behind the design。
'before_script' 选项不适合我们的工作流程，因为我们在 gitlab-ci.yml 定义中定义了特权阶段和 non-privledged 阶段的组合。 non-privileged 作业构建和 QA 代码，而特权作业打包和发布代码。 VAULT_ROLE_ID 和 VAULT_SECRET_ID 作业变量应该只对特权包和发布作业可见。
我还尝试使用 include、extend 和 yaml 锚点，但我想将项目合并到现有的 yaml 映射（script: {} 或 before_script: {}）中，而不是替换映射中的所有项目使用模板。
假设我想用 Vault 中的值创建一个变量。
variables:
$SSH_PRIVATE_KEY: `vault kv get -field=private_key project/production`
before_script:
- echo "$SSH_PRIVATE_KEY"
可以吗？
还有其他方法可以在管道内使用 Vault 机密吗？
2019 年 7 月的原始答案：
您可以看到它在 before/after 脚本步骤中使用，最后带有一个已撤销的令牌。
以 gitlab.eng.cleardata.com pub/pipelines/gcp-ci.yml 为例：
# Obtains credentials via vault (the gitlab-runner authenticates to vault using its AWS credentials)
# Configures the `gcloud` sdk and `kubectl` to authenticate to our *production* cluster
#
# Note: Do not override the before_script or the after_script in your job
#
.auth-prod: &auth-prod
image: cleardata/bionic
before_script:
- |
export CLUSTER_NAME=production
export CLUSTER_LOCATION=us-central1
export CLUSTER_PROJECT_ID=cleardata-production-cluster
- vault login -method=aws -path=gitlab-ci -no-print header_value=gitlab.eng.cleardata.com
- GCP_CREDS=$(vault read -field=private_key_data gitlab-ci/gcp/cleardata-production-cluster/key/deployment-key)
- gcloud auth activate-service-account --key-file=<(base64 -d <<<$GCP_CREDS)
- gcloud auth configure-docker
- gcloud beta container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_LOCATION --project $CLUSTER_PROJECT_ID
after_script:
- vault token revoke -self
2020 年 3 月更新：GitLab 12.9 支持此功能
HashiCorp Vault GitLab CI/CD Managed Application
GitLab wants to make it easy for users to have modern secrets management. We are now offering users the ability to install Vault within a Kubernetes cluster as part of the GitLab CI managed application process.
This will support the secure management of keys, tokens, and other secrets at the project level in a Helm chart installation.
参见 documentation and issue。
2020 年 4 月：GitLab 12.10：
Retrieve CI/CD secrets from HashiCorp Vault
In this release, GitLab adds support for lightweight JSON Web Token (JWT) authentication to integrate with your existing HashiCorp Vault.
Now, you can seamlessly provide secrets to CI/CD jobs by taking advantage of HashiCorp’s JWT authentication method rather than manually having to provide secrets as a variable in GitLab.
参见 documentation and issue。
参见 GitLab 13.4（2020 年 9 月）
仅限 Premium/Silver:
Use HashiCorp Vault secrets in CI jobs
In GitLab 12.10, GitLab introduced functionality for GitLab Runner to fetch and inject secrets into CI jobs. GitLab is now expanding the JWT Vault Authentication method by building a new
secretssyntax in the.gitlab-ci.ymlfile. This makes it easier for you to configure and use HashiCorp Vault with GitLab.https://about.gitlab.com/images/13_4/vault_ci.png -- Use HashiCorp Vault secrets in CI jobs
See Documentation and Issue.
参见 GitLab 13.9（2021 年 2 月）
Vault JWT (JSON Web Token) supports GitLab environments.
To simplify integrations with HashiCorp Vault, we’ve shipped Vault JWT token support. From the launch, you could restrict access based on data in the JWT. This release gives you a new dimension for restricting access to credentials: the environment a job targets.
This release extends the existing Vault JWT token to support environment-based restrictions too. As the
environmentname could be supplied by the user running the pipeline, we recommend you use the new environment-based restrictions with the already-existingref_typevalues for maximum security.See Documentation and Issue.
我们在构建器映像中嵌入了一个帮助脚本，它可以将 GitLab CI/CD 指向秘密的作业变量转换为包含 Vault 秘密的作业环境变量。在我们的例子中，我们还使用 appRole auth 方法来限制临时 Vault 访问令牌的有效性。
一个示例用例是：
I want a job env var "MY_SECRET_TOKEN" with a value from a Vault secret.
So I add a CI/CD variable called V_MY_SECRET_TOKEN="secret/<path>/<key>"
Then I insert a job step to retrieve the secret value and populate
the MY_SECRET_TOKEN with the value associated with the key.
在 GitLab 中添加到 CICD 作业设置的变量。
VAULT_ADDR=https://vault.example.com:8200
VAULT_ROLE_ID=db02de05-fa39-4855-059b-67221c5c2f63
VAULT_SECRET_ID=6a174c20-f6de-a53c-74d2-6018fcceff64
VAULT_VAR_FILE=/var/tmp/vault-vars.sh
添加到 .gitlab-ci.yml 作业定义的步骤。
script:
- get-vault-secrets-by-approle > ${VAULT_VAR_FILE}
- source ${VAULT_VAR_FILE} && rm ${VAULT_VAR_FILE}
这里引用了 get-vault-secrets-by-approle helper script we use.
Here is a writeup of the thinking behind the design。
'before_script' 选项不适合我们的工作流程，因为我们在 gitlab-ci.yml 定义中定义了特权阶段和 non-privledged 阶段的组合。 non-privileged 作业构建和 QA 代码，而特权作业打包和发布代码。 VAULT_ROLE_ID 和 VAULT_SECRET_ID 作业变量应该只对特权包和发布作业可见。
我还尝试使用 include、extend 和 yaml 锚点，但我想将项目合并到现有的 yaml 映射（script: {} 或 before_script: {}）中，而不是替换映射中的所有项目使用模板。