如何在 google-cloud-ml 中设置 pytorch

Question

我尝试在 google-cloud-ml 中使用 Pytorch 代码抛出作业。所以我编码 "setup.py" 文件。并添加选项 "install_requires"

"setup.py"

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl','torchvision']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My keras trainer application package.'
)

然后把作业丢给google-cloud-ml，但是没用

有错误信息

{
 insertId:  "3m78xtf9czd0u"  
 jsonPayload: {
  created:  1516845879.49039   
  levelname:  "ERROR"   
  lineno:  829   
  message:  "Command '['pip', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', u'trainer-0.1.tar.gz']' returned non-zero exit status 1"   
  pathname:  "/runcloudml.py"   
 }
 labels: {
  compute.googleapis.com/resource_id:  "6637909247101536087"   
  compute.googleapis.com/resource_name:  "cmle-training-master-5502b52646-0-ql9ds"   
  compute.googleapis.com/zone:  "us-central1-c"   
  ml.googleapis.com/job_id:  "run_ml_engine_pytorch_test_20180125_015752"   
  ml.googleapis.com/job_id/log_area:  "root"   
  ml.googleapis.com/task_name:  "master-replica-0"   
  ml.googleapis.com/trial_id:  ""   
 }
 logName:  "projects/exem-191100/logs/master-replica-0"  
 receiveTimestamp:  "2018-01-25T02:04:55.421517460Z"  
 resource: {
  labels: {…}   
  type:  "ml_job"   
 }
 severity:  "ERROR"  
 timestamp:  "2018-01-25T02:04:39.490387916Z"  
}

=========================================== =======================

See detailed message here

那么我如何在 google 云 ml 引擎中使用 pytorch？

Answer 1

实际的错误消息有点隐蔽，但它是这样的：

'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'://downl'"

要使用未托管在 PyPI 上的包，您需要使用 dependency_links（请参阅 this 文档）。像这样的东西应该可以工作：

from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['torchvision']
DEPENDENCY_LINKS = ['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    dependency_links=DEPENDENCY_LINKS,
    packages=find_packages(),
    include_package_data=True,
    description='My keras trainer application package.'
)

Answer 2

我在 google-cloud-ml

中找到了关于设置 PYTORCH 的解决方案

先你必须得到一个关于 pytorch 的 .whl 文件并将它存储到 google 存储桶。你会得到桶 link.

的 link

gs://bucketname/directory/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl

.whl 文件取决于您的 python 版本或 cuda 版本....

秒您编写命令行和 setup.py 因为您必须设置 google-cloud-ml 设置。相关 link 是这个 submit_job_to_ml-engine 您编写 setup.py 文件来描述您的设置。相关的link是这个write_setup.py_file

这是我的命令代码和setup.py文件

=========================================== ======================== "command"

#commandline code
JOB_NAME="run_ml_engine_pytorch_test_$(date +%Y%m%d_%H%M%S)"
REGION=us-central1
OUTPUT_PATH=gs://yourbucket
gcloud ml-engine jobs submit training $JOB_NAME \
    --job-dir $OUTPUT_PATH \
    --runtime-version 1.4 \
    --module-name models.pytorch_test \
    --package-path models/ \
    --packages gs://yourbucket/directory/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl \
    --region $REGION \
    -- \
    --verbosity DEBUG

=========================================== ======================== "setup.py"

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['torchvision']
setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My pytorch trainer application package.'
)

=========================================== ========================

第三如果您有向 ml-engine 提交作业的经验。你可能知道关于提交 ml-engine 的文件结构 packaging_training_model。你必须按照上面的link并知道如何打包文件。

如何在 google-cloud-ml 中设置 pytorch

How to setup pytorch in google-cloud-ml

google-cloud-ml

pytorch

有错误信息