apache camel 从 s3 读取文件的问题
Problem with apache camel reading a file from s3
我正在实施一条路线，从 s3 读取文件并将其删除。问题是删除后继续搜索文件出错，不明白路由为什么不停
这是我的
var uri = "aws-s3://$s3BucketName?amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true"
from(uri)
2021-02-23 23:53:23.226 WARN 2452 --- [ftp-bucket/test] o.a.camel.component.aws.s3.S3Consumer : Consumer S3Consumer[aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV] failed polling endpoint: aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV. Will try again at next poll. Caused by: [com.amazonaws.services.s3.model.AmazonS3Exception - The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=)]
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access0(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.714.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) ~[aws-java-sdk-s3-1.11.714.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) ~[aws-java-sdk-s3-1.11.714.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486) ~[aws-java-sdk-s3-1.11.714.jar:na]
at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:74) ~[camel-aws-s3-3.2.0.jar:3.2.0]
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) ~[camel-support-3.2.0.jar:3.2.0]
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106) ~[camel-support-3.2.0.jar:3.2.0]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
您使用的是消费者，因此它将定期轮询。您有多种选择来解决问题：
在您的 S3 端点上使用 repeatCount 选项
from("aws-s3://$s3BucketName amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true&repeatCount=1")
使用定时器+生产者
生产者端的AWS-S3有deleteObject操作[1]
所以你可以这样做：
from(timer)
to(aws-s3)
并指定 deleteObject 操作所需的选项和 headers。
重要的一点是，camel 3.2.0 是一个开发版本。我们建议使用 3.7.x 系列的最后一个受支持的 LTS 版本，即 3.7.2.
还有一点，AWS-S3组件在3.8.0已经弃用，3.9.0将被移除，所以请切换到Camel-AWS2-S3。
我正在实施一条路线，从 s3 读取文件并将其删除。问题是删除后继续搜索文件出错，不明白路由为什么不停
这是我的
var uri = "aws-s3://$s3BucketName?amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true"
from(uri)
2021-02-23 23:53:23.226 WARN 2452 --- [ftp-bucket/test] o.a.camel.component.aws.s3.S3Consumer : Consumer S3Consumer[aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV] failed polling endpoint: aws-s3://bucket/test?amazonS3Client=%23s3Client&deleteAfterRead=true&fileName=testAlex.CSV. Will try again at next poll. Caused by: [com.amazonaws.services.s3.model.AmazonS3Exception - The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=)] com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: ; S3 Extended Request ID: /=) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1742) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1371) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1347) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access0(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) ~[aws-java-sdk-s3-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) ~[aws-java-sdk-s3-1.11.714.jar:na] at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486) ~[aws-java-sdk-s3-1.11.714.jar:na] at org.apache.camel.component.aws.s3.S3Consumer.poll(S3Consumer.java:74) ~[camel-aws-s3-3.2.0.jar:3.2.0] at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) ~[camel-support-3.2.0.jar:3.2.0] at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106) ~[camel-support-3.2.0.jar:3.2.0] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
您使用的是消费者，因此它将定期轮询。您有多种选择来解决问题：
在您的 S3 端点上使用 repeatCount 选项
from("aws-s3://$s3BucketName amazonS3Client=#s3Client&fileName=$fileName&deleteAfterRead=true&repeatCount=1")
使用定时器+生产者 生产者端的AWS-S3有deleteObject操作[1]
所以你可以这样做：
from(timer)
to(aws-s3)
并指定 deleteObject 操作所需的选项和 headers。
重要的一点是，camel 3.2.0 是一个开发版本。我们建议使用 3.7.x 系列的最后一个受支持的 LTS 版本，即 3.7.2.
还有一点，AWS-S3组件在3.8.0已经弃用，3.9.0将被移除，所以请切换到Camel-AWS2-S3。