ML 引擎因作曲家失败 - 未知名称 "python-version"

Question

我正在尝试使用 cloud composer 启动 ml-engine jobs submit training，我正在使用本指南获取说明 recommendation-system-tensorflow-deploy。

我正在使用 google 创建的插件 (see the implementation here)

我正在尝试使其在 python 版本 3.5 上运行，方法是将第 206 行从：

更改为

training_request = {
        'jobId': job_id,
        'trainingInput': {
            'scaleTier': self._scale_tier,
            'packageUris': self._package_uris,
            'pythonModule': self._training_python_module,
            'region': self._region,
            'args': self._training_args,
            'masterType': self._master_type
        }

收件人：

training_request = {
    'jobId': job_id,
    'trainingInput': {
        'scaleTier': self._scale_tier,
        'packageUris': self._package_uris,
        'pythonModule': self._training_python_module,
        'region': self._region,
        'args': self._training_args,
        'masterType': self._master_type,
        'python-version': '3.5'     #self._python_version

    }

我也尝试向其中添加运行时间版本 (runtime-version='1.12')，但我不断收到以下错误：

[2019-01-20 11:58:36,331] {models.py:1594} ERROR - <HttpError 400 when requesting https://ml.googleapis.com/v1/projects/hallowed-forge-577/jobs?alt=json returned "Invalid JSON payload received. Unknown name "python-version" at 'job.training_input': Cannot find field.">
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models.py", line 1492, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/home/airflow/gcs/plugins/ml_engine_plugin.py", line 241, in execut
    self._project_id, training_request, check_existing_job
  File "/home/airflow/gcs/plugins/ml_engine_plugin.py", line 79, in create_jo
    request.execute(
  File "/usr/local/lib/python3.6/site-packages/oauth2client/util.py", line 135, in positional_wrappe
    return wrapped(*args, **kwargs
  File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 838, in execut
    raise HttpError(resp, content, uri=self.uri
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://ml.googleapis.com/v1/projects/hallowed-forge-577/jobs?alt=json returned "Invalid JSON payload received. Unknown name "python-version" at 'job.training_input': Cannot find field."
[2019-01-20 11:58:36,334] {models.py:1623} INFO - Marking task as FAILED.
[2019-01-20 11:58:36,513] {models.py:1627} ERROR - Failed to send email to: ['airflow@example.com']
[2019-01-20 11:58:36,516] {models.py:1628} ERROR - HTTP Error 401: Unauthorized
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models.py", line 1625, in handle_failur
    self.email_alert(error, is_retry=False
  File "/usr/local/lib/airflow/airflow/models.py", line 1778, in email_aler
    send_email(task.email, title, body
  File "/usr/local/lib/airflow/airflow/utils/email.py", line 44, in send_emai
    return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype
  File "/usr/local/lib/airflow/airflow/contrib/utils/sendgrid.py", line 116, in send_emai
    _post_sendgrid_mail(mail.get()
  File "/usr/local/lib/airflow/airflow/contrib/utils/sendgrid.py", line 122, in _post_sendgrid_mai
    response = sg.client.mail.send.post(request_body=mail_data
  File "/usr/local/lib/python3.6/site-packages/python_http_client/client.py", line 252, in http_reques
    return Response(self._make_request(opener, request, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/python_http_client/client.py", line 176, in _make_reques
    raise ex
python_http_client.exceptions.UnauthorizedError: HTTP Error 401: Unauthorize

请注意，python 版本实际上发生了变化（从原来的 2.7 变为 3.6），因此更改 python 版本会有所作为，但随后会卡住

任何关于我在这里遗漏的帮助都会很棒！

Answer 1

该示例似乎使用了旧版本的 airflow MLEngineTrainingOperator。最后一个版本实现了 runtime-version/python-version 训练参数。使用当前版本： mlengine_operator.py

ML 引擎因作曲家失败 - 未知名称 "python-version"

ML-engine fails from composer -Unknown name "python-version"

python

google-cloud-ml

google-cloud-composer