通过 SDK 启用 AWS EMRFS Consistent View

Question

通常通过 emrfs-site.xml 启用 emrfs 一致性 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emrfs-configure-consistent-view.html

有谁知道是否可以通过 SDK 访问这些设置？

Answer 1

是的，这里有完整的文档： http://docs.aws.amazon.com/ElasticMapReduce/latest/API/Welcome.html

您需要首先授权连接到您的 AWS，然后您可以使用 API 根据您的需要配置您的应用程序。也看看这里： http://docs.aws.amazon.com/ElasticMapReduce/latest/API/CommonParameters.html http://docs.aws.amazon.com/ElasticMapReduce/latest/API/EmrConfigurations.html

Answer 2

要使用 Java SDK 启用 EMRFS，需要将 "emrfs-site" 配置添加到 RunJobFlowRequest 和 fs.s3.consistent 属性必须设置为 true。像这样：

Map<String, String> emrfsProperties = new HashMap<>();
emrfsProperties.put("fs.s3.consistent", "true");

RunJobFlowRequest request = new RunJobFlowRequest()
        ....
        .withServiceRole(SERVICE_ROLE)
        .withJobFlowRole(JOB_FLOW_ROLE)
        .withConfigurations(
                new Configuration().withClassification("yarn-site").withProperties(yarnProperties),
                new Configuration().withClassification("emrfs-site").withProperties(emrfsProperties)
        )
        .withInstances(new JobFlowInstancesConfig()
                .withEc2KeyName(EC2_KEYPAIR)
                ....

可以找到 EMRFS 配置参数的完整列表 here

通过 SDK 启用 AWS EMRFS Consistent View

AWS EMRFS Consistent View enable via SDK

amazon-web-services

emr

aws-sdk