ECS 中的 Flink 无法找到 shaded ContainerCredentialsProvider

Question

我正在使用 Fargate 在 ECS 上尝试运行 Flink 1.7.2。我已经将我的工作的状态后端设置为 RocksDB，路径为 s3://...

在我的 Dockerfile 中，我的基本映像是 1.7.2-hadoop27-scala_2.11，我运行以下 2 个命令：

RUN echo "fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"
RUN cp /opt/flink/opt/flink-s3-fs-hadoop-1.7.2.jar /opt/flink/lib/flink-s3-fs-hadoop-1.7.2.jar

就像里面说的 https://issues.apache.org/jira/browse/FLINK-8439

但是我得到以下异常：

Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:592)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:556)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:256)
        at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125)
        at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395)
        at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
        at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
        at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407)
        at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
        ... 17 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2375)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2446)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:589)
        ... 28 more

查看 flink-s3-fs-hadoop-1.7.2.jar 我看到 class ContainerCredentialsProvider 的包实际上是 org.apache.flink.fs.s3base.shaded.com.amazonaws.auth

我已经试过了：

将 aws-sdk-core jar 添加到库中，并将凭据提供程序设置为 com.amazonaws.auth.ContainerCredentialsProvider（没有阴影），但我遇到了上面问题 link 中提到的问题
将凭据提供程序设置为 org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.ContainerCredentialsProvider 但 S3FileSystemFactory.java 中的代码会在其前面加上 org.apache.flink.fs.s3hadoop.shaded.

这里有找到 class 的任何想法吗？

Answer 1

此问题已在之后的其中一个版本中得到解决。

我运行它在 Flink 1.9.0 集群上具有以下行：

RUN echo "fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"

并且找到 class 并且有效。

您可以看到： https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java

FLINK_SHADING_PREFIX 现在是正确的

ECS 中的 Flink 无法找到 shaded ContainerCredentialsProvider

Flink in ECS fails to find shaded ContainerCredentialsProvider

hadoop

amazon-s3

apache-flink

flink-streaming