如何在 Bluemix 对象存储中转储 joblib 或 pickle 文件？

Question

我正在 Bluemix 上使用 Python 应用程序和 Flask 运行。我知道如何将对象存储与 swiftclient 模块一起使用来创建容器并在其中保存文件，但是如何转储其中包含的 joblib 或 pickle 文件？我如何将它加载回我的 Python 程序？

下面是存储简单文本文件的代码。

import swiftclient

app = Flask(__name__)
CORS(app)


cloudant_service = json.loads(os.environ['VCAP_SERVICES'])['Object-Storage'][0]
objectstorage_creds = cloudant_service['credentials']

if objectstorage_creds:
   auth_url = objectstorage_creds['auth_url'] + '/v3' #authorization URL
   password = objectstorage_creds['password'] #password
   project_id = objectstorage_creds['projectId'] #project id
   user_id = objectstorage_creds['userId'] #user id 
   region_name = objectstorage_creds['region'] #region name 

def predict_joblib():
  print('satart')
  conn = swiftclient.Connection(key=password,authurl=auth_url,auth_version='3',os_options={"project_id": project_id,"user_id": user_id,"region_name": region_name})
  container_name = 'new-container'

  # File name for testing
  file_name = 'requirment.txt'

  # Create a new container
  conn.put_container(container_name)
  print ("nContainer %s created successfully." % container_name)

  # List your containers
  print ("nContainer List:")
  for container in conn.get_account()[1]:
    print (container['name'])

  # Create a file for uploading
  with open(file_name, 'w') as example_file:
    conn.put_object(container_name,file_name,contents= "",content_type='text/plain')

  # List objects in a container, and prints out each object name, the file size, and last modified date
  print ("nObject List:")
  for container in conn.get_account()[1]:
    for data in conn.get_container(container['name'])[1]:
      print ('object: {0}t size: {1}t date: {2}'.format(data['name'], data['bytes'], data['last_modified']))

  # Download an object and save it to ./my_example.txt
  obj = conn.get_object(container_name, file_name)
  with open(file_name, 'w') as my_example:
    my_example.write(obj[1])
  print ("nObject %s downloaded successfully." % file_name)




@app.route('/')
def hello():
    dff = predict_joblib()
    return 'Welcome to Python Flask!'

@app.route('/signUp')
def signUp():
    return 'signUp'


port = os.getenv('PORT', '5000')
if __name__ == "__main__":
    app.debug = True
    app.run(host='0.0.0.0', port=int(port))

Answer 1

由于 file.open 和 pickle.dumps returns 字节对象都出现在 python 文档中：

pickle.dumps(obj, protocol=None, *, fix_imports=True) Return 对象的腌制表示形式为字节对象，而不是将其写入文件。

打开（名称[，模式[，缓冲]]） 打开文件，返回文件对象部分中描述的文件类型的对象。如果无法打开文件，则会引发 IOError。打开文件时，最好使用 open() 而不是直接调用文件构造函数。

您可以只处理要存储为 obj 的对象，例如：

# Create a file for uploading
file = pickle.dumps(obj)
conn.put_object(container_name,file,contents= "",content_type='application/python-pickle')

这种内容类型的变化是由于 http 协议中的标准所致。这是我从另一个 SO 问题中得到的，请检查。如所述：

It is the de-facto standard. RFC2046 states: 4.5.3. Other Application Subtypes It is expected that many other subtypes of "application" will be defined in the future. MIME implementations must at a minimum treat any unrecognized subtypes as being equivalent to "application/octet- stream". So, to a non-pickle-aware system, the stream will look like any other octet-stream, but for a pickle-enabled system this is vital information

如何在 Bluemix 对象存储中转储 joblib 或 pickle 文件？

How do I dump a joblib or pickle file in Bluemix Object Storage?

python

pickle

object-storage

ibm-cloud