Python Boto3 put_object 来自 s3 中的 lambda 的文件

Question

我想通过 lambda 在 s3 中发送一个 json 文件。我在文档中看到我们可以使用函数 boto3 put_object 发送文件或字节对象 (Body=b'bytes'|file).

但如果我没记错的话，如果我在 s3 中发送 Body=bytes 的文件，然后下载我的文件，内容将不可见。

所以在我的 lambda 函数中，我从 SQS 队列接收消息，我在 lambda 临时文件夹 /tmp 中创建了一个包含消息内容的文件。我想获取此 json 文件以将其发送到 my_bucket/folder/file.json

我看到很多在 s3 中创建文件的示例，但 Body 参数以字节为单位而不是文件。

这是我的代码 (python3.7)

def alpaca_consent_customer_dev(event, context):  # handler
    # TODO implement
    request_id = context.aws_request_id
    print('START - RequestID: {}'.format(request_id))

    # function to write json file
    def write_json(target_path, target_file, data):
        if not os.path.exists(target_path):
            try:
                os.makedirs(target_path)
            except Exception as e:
                print(e)
                raise
        with open(os.path.join(target_path, target_file), 'w') as f:
            json.dump(data, f)

    try:
        s3 = boto3.client('s3', region_name="us-west-2")

        request_id = context.aws_request_id
        print('START - RequestID: {}'.format(request_id))
        
         # Get message from SQS queue
        for record in event['Records']:

            data = record
            
            # Get message from SQS
            data_loaded = json.loads(data['body'])

            sns_message_id = data_loaded['MessageId']

            print('data loaded type:', type(data_loaded))

            data_saved = json.dumps(data_loaded)

            # Create json file in temporary folder
            write_json('/tmp', sns_message_id+'.json', data_saved)

            # Check if file exists
            print(glob.glob("/tmp/*.json"))
            # result: ['/tmp/3bb1c0bc-68d5-5c4d-b827-021301.json']


            s3.put_object(Body='/tmp/'+sns_message_id + '.json', Bucket='mybucket', Key='my_sub_bucket/' + datetime.datetime.today().strftime('%Y%m%d')+ '/'+ sns_message_id + '.json')
    
    except Exception as e:
        raise Exception('ERROR lambda failed: {}'.format(str(e)))

感谢您的帮助。问候。

Answer 1

有一个 official example in the boto3 docs:

import logging
import boto3
from botocore.exceptions import ClientError


def upload_file(file_name, bucket, object_name=None):
    """Upload a file to an S3 bucket

    :param file_name: File to upload
    :param bucket: Bucket to upload to
    :param object_name: S3 object name. If not specified then file_name is used
    :return: True if file was uploaded, else False
    """

    # If S3 object_name was not specified, use file_name
    if object_name is None:
        object_name = file_name

    # Upload the file
    s3_client = boto3.client('s3')
    try:
        response = s3_client.upload_file(file_name, bucket, object_name)
    except ClientError as e:
        logging.error(e)
        return False
    return True

你可以只使用s3客户端的upload_file method。

Python Boto3 put_object 来自 s3 中的 lambda 的文件

Python Boto3 put_object file from lambda in s3

amazon-s3

amazon-sqs

python-3.x

aws-lambda