如何使用 Python SDK 将大文件 (~100mb) 上传到 Azure blob 存储？

Question

我正在使用最新的 Azure 存储 SDK (azure-storage-blob-12.7.1)。它适用于较小的文件，但对于大于 30MB 的较大文件会抛出异常。

azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

from azure.storage.blob import BlobServiceClient, PublicAccess, BlobProperties,ContainerClient

    def upload(file):
        settings = read_settings()
        connection_string = settings['connection_string']
        container_client = ContainerClient.from_connection_string(connection_string,'backup')
        blob_client = container_client.get_blob_client(file)
        with open(file,"rb") as data:
            blob_client.upload_blob(data)
            print(f'{file} uploaded to blob storage')
    
    upload('crashes.csv')

Answer 1

当我尝试上传约 180MB 的 .txt 文件时，您的代码似乎对我来说一切正常。但是，如果上传小文件适合您，我认为将大文件分成小部分上传可能是一种解决方法。试试下面的代码：

from azure.storage.blob import BlobClient

storage_connection_string=''
container_name = ''
dest_file_name = ''

local_file_path = ''

blob_client = BlobClient.from_connection_string(storage_connection_string,container_name,dest_file_name)

#upload 4 MB for each request
chunk_size=4*1024*1024  

if(blob_client.exists):
    blob_client.delete_blob()
    blob_client.create_append_blob()

with open(local_file_path, "rb") as stream:
    
    while True:
            read_data = stream.read(chunk_size)
            
            if not read_data:
                print('uploaded')
                break 
            blob_client.append_block(read_data)

结果：

如何使用 Python SDK 将大文件 (~100mb) 上传到 Azure blob 存储？

How to upload large file (~100mb) to Azure blob storage using Python SDK?

azure-storage

azure-blob-storage

azure-sdk-python