AWS Glue 爬虫访问被拒绝并附加了 AmazonS3FullAccess

Question

我刚刚设置了一个 AWS Glue 爬虫来爬取 S3 存储桶。我为爬虫设置了一个 IAM 角色，并将托管策略 "AWSGlueServiceRole" 和 "AmazonS3FullAccess" 附加到该角色。我已确保爬虫正在使用该角色。但是，每次我运行爬虫我都会在日志中收到类似这样的错误消息：

ERROR : Error Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <omitted>; S3 Extended Request ID: <omitted>) retrieving file at s3://my-bucket/snapshots/snapshot-1/mydb/mydb.mytable/11/part-00000-ffffffff-ffff-ffff-ffff-ffffffffffff-c000.gz.parquet. Tables created did not infer schemas from this file.

我已确认其执行角色附加了 "AmazonS3ReadOnlyAccess" 的 Lambda 能够访问存储桶。我做错了什么？

编辑：设置 "block all public access" 或禁用它没有明显效果。

EDIT2：IAM 角色的托管策略文档如下。没有内联政策。

AWSGlueServiceRole:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:*",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketAcl",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeRouteTables",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcAttribute",
                "iam:ListRolePolicies",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "cloudwatch:PutMetricData"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket"
            ],
            "Resource": [
                "arn:aws:s3:::aws-glue-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::aws-glue-*/*",
                "arn:aws:s3:::*/*aws-glue-*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::crawler-public*",
                "arn:aws:s3:::aws-glue-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:/aws-glue/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": [
                        "aws-glue-service-resource"
                    ]
                }
            },
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*",
                "arn:aws:ec2:*:*:security-group/*",
                "arn:aws:ec2:*:*:instance/*"
            ]
        }
    ]
}

AmazonS3FullAccess:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

Answer 1

原来是 KMS 问题。该存储桶包含一个 Aurora RDS 快照的导出，并且该快照显然是加密写入的。所以一旦我添加了以下策略，我就设置好了：

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt"
    ],
    "Resource": [
      "arn:aws:kms:<region>:<my account id>:key/<my key id>"
    ]
  }
}

这是我附加到该角色的整个托管策略（请注意该角色还附加了 AWSGlueServiceRole）：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/snapshots*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:<region>:<my account id>:key/<my key id>"
            ]
        }
    ]
}

AWS Glue 爬虫访问被拒绝并附加了 AmazonS3FullAccess

AWS Glue Crawler Access Denied with AmazonS3FullAccess attached

amazon-s3

amazon-web-services

aws-glue