如何使用 curl 将 .gitlab-ci.yml 内容的 linting 请求 post 发送到 gitlab api？
How to use curl to post a linting request with the contents of .gitlab-ci.yml to the gitlab api?
尝试向 gitlab.com api 发出 linting .gitlab-ci.yaml 文件的 curl 请求，但收到错误的请求响应：{"status":400,"error":"Bad Request"}
#!/usr/bin/env bash
PAYLOAD=$( cat << JSON
{ "content":
$(<$PWD/../.gitlab-ci.yml)
JSON
)
echo "Payload is $PAYLOAD"
curl --include --show-error --request POST --header "Content-Type: application/json" --header "Accept: application/json" "https://gitlab.com/api/v4/ci/lint" --data-binary "$PAYLOAD"
有没有人设法通过 bash 脚本成功地检查 .gitlab-ci.yml？还尝试将内容有效负载包装在大括号中并收到相同的响应。
更新
我认为发生的事情是 GitLab CI 端点期望将 .gitlab-ci yaml 文件的内容转换为 json 以用于 POST 要求。参见 here
修改脚本以在发送前使用 ruby 将 yaml 转换为 json，这适用于简单的 .gitlab-ci.yml。但是，当我的项目使用 yaml 文件时，出现错误：{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 221"]}% 当我使用 gitlab 网页进行 linting 时，文件有效。
{"content": "{ \"stages\": [ \"build\", \"test\", \"pages\", \"release\" ], \"variables\": { \"DOCKER_DRIVER\": \"overlay2\" }, \"services\": [ \"docker:19.03.11-dind\" ], \"build:plugin\": { \"image\": \"docker:19.03.11\", \"stage\": \"build\", \"before_script\": [ \"echo \"$CI_JOB_TOKEN\" | docker login -u gitlab-ci-token --password-stdin \"$CI_REGISTRY\"\" ].....
第 221 列是上述 json 摘录中的 \"image\": \"docker:19.03.11\"，特别是 ci 结束转义引号。认为是引号转义不正确的问题？？
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < .gitlab-ci.yml)
# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\)"/\"/g')
# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'
echo "${json_content}"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
第二次更新
使用上面的 bash 编写此 yaml 文件的脚本：
stages:
- test
test:
stage: test
script:
- echo "test"
转换为 json:
{"content": "{ \"stages\": [ \"test\" ], \"test\": { \"stage\": \"test\", \"script\": [ \"echo \"test\"\" ] } }"}
发送到 api 时收到以下 json 错误响应：
{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 62"]}%
使用以下脚本终于成功了：
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' custom_hooks/valid.yml)
# escape quotes
json_content=$(echo $json | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))')
echo $json_content
# Add object contect for GitLab linter
json_content="{\"content\": ${json_content}}"
# Output escaped content to file
echo $json_content > custom_hooks/input.json
echo "Escaped json content written to file input.json"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
N.B 将调整脚本以从系统参数而不是固定文件位置 custom_hooks/valid.yml 读取文件。此外，JSON 响应需要使用 jq 或 python / ruby 命令 shell 进行解析。包括这个脚本，希望它能帮助其他人。
问题是最初我将文件的 YAML 内容直接发送到 api:
{ "content": { <contents of .gitlab-yml> } }
看起来 GitLab 接受在其 API 中转换为转义 JSON 字符串的 YAML。因此使用 ruby 将 yaml 转换为 JSON，然后使用 python 转义 ruby 生成的结果 JSON。终于能够使用 curl 将转义的 JSON 字符串发送到 GitLab API 进行验证.....
不确定 Ruby 是否有等同于 python 的 json.dumps ....但这个解决方案允许我验证 gitlab-ci...下一阶段连接到 git 预提交挂钩/服务器端预接收（如果可能！）以防止无效的 .gitlab-ci.yml 文件中断 CI管道。
ruby 的新手...自从发布原始答案以来，我们已经着手创建一个 ruby 脚本，可以从预提交挂钩等中使用。现在只需要 bash 和 ruby:
#!/usr/bin/env ruby
require 'json'
require 'net/http'
require 'optparse'
require 'yaml'
=begin
POST to GitLab api for linting ci yaml
Params:
+url+ :: Api url
+yaml+ :: Yaml payload for linting
Returns:
Json validation result from API for HTTP response Success
Aborts with HTTP Message for all other status codes
=end
def call_api(url, yaml)
uri = URI.parse(url)
req = Net::HTTP::Post.new(uri)
req.content_type='application/json'
req['Accept']='application/json'
req.body = JSON.dump({"content" => yaml.to_json})
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_PEER
response = https.request(req)
case response
when Net::HTTPSuccess
puts "request successful"
return JSON.parse response.body
when Net::HTTPUnauthorized
abort("#{response.message}: invalid token in api request?")
when Net::HTTPServerError
abort('error' => "#{response.message}: server error, try again later?")
when Net::HTTPBadRequest
puts "Bad request..." + request.body
abort("#{response.message}: bad api request?")
when Net::HTTPNotFound
abort("#{response.message}: api request not found?")
else
puts "Failed validation\nJSON payload :: #{request.body}\nHTTP Response: #{response.message}"
abort("#{response.message}: failed api request?")
end
end
=begin
Display exit report and raise the appropriate system exit code
Params:
+status+ :: Validation status string. Legal values are valid or invalid
+errors+ :: String array storing errors if yaml was reported as invalid
Returns:
Exits with 0 when successful
Exits with 1 on validation errors or fails to parse legal status value
=end
def exit_report(status, errors)
case status
when "valid"
puts ".gitlab-ci.yml is valid"
exit(0)
when "invalid"
abort(".gitlab-ci.yml is invalid with errors:\n\n" + errors.join("\n"))
else
abort("A problem was encountered parsing status : " + status)
end
end
=begin
Load yaml file from path and return contents
Params:
+path+ :: Absolute or relative path to .gitlab-ci.yml file
=end
def load_yaml(path)
begin
YAML.load_file(path)
rescue Errno::ENOENT
abort("Failed to load .gitlab-ci.yml")
end
end
=begin
Parse command line options
Returns:
Hash containing keys: {:yaml_file,:url}
=end
def read_args()
options = {}
OptionParser.new do |opt|
opt.on('-f', '--yaml YAML-PATH', 'Path to .gitlab-ci.yml') { |o| options[:yaml_file] = o }
opt.on('-l', '--url GitLab url', 'GitLab API url') { |o| options[:url] = o }
end.parse!
options
end
=begin
Load yaml to send to GitLab API for linting
Display report of linting retrieved from api
Returns:
Exits with 0 upon success and 1 when errors encountered
=end
def main()
# try and parse the arguments
options = read_args()
unless !options.has_key?(:yaml_file) || !options.has_key?(:url)
# try and load the yaml from path
puts "Loading file #{options[:yaml_file]}"
yaml = load_yaml(options[:yaml_file])
# make lint request to api
puts "Making POST request to #{options[:url]}"
response_data=call_api(options[:url], yaml)
# display exit report and raise appropriate exit code
unless !response_data.has_key?("status") || !response_data.has_key?("errors")
exit_report response_data["status"], response_data["errors"]
else
puts "Something went wrong parsing the json response " + response_data
end
else
abort("Missing required arguments yaml_file and url, use -h for usage")
end
end
# start
main
尝试向 gitlab.com api 发出 linting .gitlab-ci.yaml 文件的 curl 请求，但收到错误的请求响应：{"status":400,"error":"Bad Request"}
#!/usr/bin/env bash
PAYLOAD=$( cat << JSON
{ "content":
$(<$PWD/../.gitlab-ci.yml)
JSON
)
echo "Payload is $PAYLOAD"
curl --include --show-error --request POST --header "Content-Type: application/json" --header "Accept: application/json" "https://gitlab.com/api/v4/ci/lint" --data-binary "$PAYLOAD"
有没有人设法通过 bash 脚本成功地检查 .gitlab-ci.yml？还尝试将内容有效负载包装在大括号中并收到相同的响应。
更新
我认为发生的事情是 GitLab CI 端点期望将 .gitlab-ci yaml 文件的内容转换为 json 以用于 POST 要求。参见 here
修改脚本以在发送前使用 ruby 将 yaml 转换为 json，这适用于简单的 .gitlab-ci.yml。但是，当我的项目使用 yaml 文件时，出现错误：{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 221"]}% 当我使用 gitlab 网页进行 linting 时，文件有效。
{"content": "{ \"stages\": [ \"build\", \"test\", \"pages\", \"release\" ], \"variables\": { \"DOCKER_DRIVER\": \"overlay2\" }, \"services\": [ \"docker:19.03.11-dind\" ], \"build:plugin\": { \"image\": \"docker:19.03.11\", \"stage\": \"build\", \"before_script\": [ \"echo \"$CI_JOB_TOKEN\" | docker login -u gitlab-ci-token --password-stdin \"$CI_REGISTRY\"\" ].....
第 221 列是上述 json 摘录中的 \"image\": \"docker:19.03.11\"，特别是 ci 结束转义引号。认为是引号转义不正确的问题？？
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))' < .gitlab-ci.yml)
# escape quotes
json_content=$(echo $json | perl -pe 's/(?<!\)"/\"/g')
# Add object contect for GitLab linter
json_content='{"content": "'${json_content}'"}'
echo "${json_content}"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
第二次更新
使用上面的 bash 编写此 yaml 文件的脚本：
stages:
- test
test:
stage: test
script:
- echo "test"
转换为 json:
{"content": "{ \"stages\": [ \"test\" ], \"test\": { \"stage\": \"test\", \"script\": [ \"echo \"test\"\" ] } }"}
发送到 api 时收到以下 json 错误响应：
{"status":"invalid","errors":["(\u003cunknown\u003e): did not find expected ',' or ']' while parsing a flow sequence at line 1 column 62"]}%
使用以下脚本终于成功了：
#!/usr/bin/env bash
json=$(ruby -ryaml -rjson -e 'puts(YAML.load(ARGF.read).to_json)' custom_hooks/valid.yml)
# escape quotes
json_content=$(echo $json | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))')
echo $json_content
# Add object contect for GitLab linter
json_content="{\"content\": ${json_content}}"
# Output escaped content to file
echo $json_content > custom_hooks/input.json
echo "Escaped json content written to file input.json"
curl --include --show-error --request POST \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
"https://gitlab.com/api/v4/ci/lint" \
--data-binary "$json_content"
N.B 将调整脚本以从系统参数而不是固定文件位置 custom_hooks/valid.yml 读取文件。此外，JSON 响应需要使用 jq 或 python / ruby 命令 shell 进行解析。包括这个脚本，希望它能帮助其他人。
问题是最初我将文件的 YAML 内容直接发送到 api:
{ "content": { <contents of .gitlab-yml> } }
看起来 GitLab 接受在其 API 中转换为转义 JSON 字符串的 YAML。因此使用 ruby 将 yaml 转换为 JSON，然后使用 python 转义 ruby 生成的结果 JSON。终于能够使用 curl 将转义的 JSON 字符串发送到 GitLab API 进行验证.....
不确定 Ruby 是否有等同于 python 的 json.dumps ....但这个解决方案允许我验证 gitlab-ci...下一阶段连接到 git 预提交挂钩/服务器端预接收（如果可能！）以防止无效的 .gitlab-ci.yml 文件中断 CI管道。
ruby 的新手...自从发布原始答案以来，我们已经着手创建一个 ruby 脚本，可以从预提交挂钩等中使用。现在只需要 bash 和 ruby:
#!/usr/bin/env ruby
require 'json'
require 'net/http'
require 'optparse'
require 'yaml'
=begin
POST to GitLab api for linting ci yaml
Params:
+url+ :: Api url
+yaml+ :: Yaml payload for linting
Returns:
Json validation result from API for HTTP response Success
Aborts with HTTP Message for all other status codes
=end
def call_api(url, yaml)
uri = URI.parse(url)
req = Net::HTTP::Post.new(uri)
req.content_type='application/json'
req['Accept']='application/json'
req.body = JSON.dump({"content" => yaml.to_json})
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_PEER
response = https.request(req)
case response
when Net::HTTPSuccess
puts "request successful"
return JSON.parse response.body
when Net::HTTPUnauthorized
abort("#{response.message}: invalid token in api request?")
when Net::HTTPServerError
abort('error' => "#{response.message}: server error, try again later?")
when Net::HTTPBadRequest
puts "Bad request..." + request.body
abort("#{response.message}: bad api request?")
when Net::HTTPNotFound
abort("#{response.message}: api request not found?")
else
puts "Failed validation\nJSON payload :: #{request.body}\nHTTP Response: #{response.message}"
abort("#{response.message}: failed api request?")
end
end
=begin
Display exit report and raise the appropriate system exit code
Params:
+status+ :: Validation status string. Legal values are valid or invalid
+errors+ :: String array storing errors if yaml was reported as invalid
Returns:
Exits with 0 when successful
Exits with 1 on validation errors or fails to parse legal status value
=end
def exit_report(status, errors)
case status
when "valid"
puts ".gitlab-ci.yml is valid"
exit(0)
when "invalid"
abort(".gitlab-ci.yml is invalid with errors:\n\n" + errors.join("\n"))
else
abort("A problem was encountered parsing status : " + status)
end
end
=begin
Load yaml file from path and return contents
Params:
+path+ :: Absolute or relative path to .gitlab-ci.yml file
=end
def load_yaml(path)
begin
YAML.load_file(path)
rescue Errno::ENOENT
abort("Failed to load .gitlab-ci.yml")
end
end
=begin
Parse command line options
Returns:
Hash containing keys: {:yaml_file,:url}
=end
def read_args()
options = {}
OptionParser.new do |opt|
opt.on('-f', '--yaml YAML-PATH', 'Path to .gitlab-ci.yml') { |o| options[:yaml_file] = o }
opt.on('-l', '--url GitLab url', 'GitLab API url') { |o| options[:url] = o }
end.parse!
options
end
=begin
Load yaml to send to GitLab API for linting
Display report of linting retrieved from api
Returns:
Exits with 0 upon success and 1 when errors encountered
=end
def main()
# try and parse the arguments
options = read_args()
unless !options.has_key?(:yaml_file) || !options.has_key?(:url)
# try and load the yaml from path
puts "Loading file #{options[:yaml_file]}"
yaml = load_yaml(options[:yaml_file])
# make lint request to api
puts "Making POST request to #{options[:url]}"
response_data=call_api(options[:url], yaml)
# display exit report and raise appropriate exit code
unless !response_data.has_key?("status") || !response_data.has_key?("errors")
exit_report response_data["status"], response_data["errors"]
else
puts "Something went wrong parsing the json response " + response_data
end
else
abort("Missing required arguments yaml_file and url, use -h for usage")
end
end
# start
main