Camel s3 consumer polling the same file every time

[Akram <ak...@gmail.com>]

I am polling files from S3 using camel aws-s3 component. But it is
downloading the files which has been already downloaded. Is there any way to
avoid this behavior and keep track of files which has been already
downloaded. Then in the next poll, it should download which are newly
created in s3.



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-s3-consumer-polling-the-same-file-every-time-tp5784157.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel s3 consumer polling the same file every time

[Raghav <ra...@cognizant.com>]

Hi,

I'm new to apache camel framework and i'm using *2.15.1* version.
My use case is to poll aws s3 bucket and download newly created files from
it and for that i'm doing as follows

*camel-route*
from("aws-s3://" + Constants.S3_BUCKET +
"?amazonS3Client=#client&deleteAfterRead=false")
        .routeId("awsFileCopyRoute")
        .log(LoggingLevel.INFO, "awsFileCopyRoute", "Camel AWS Key:
${header.CamelAwsS3Key}")
        .to("file://" + Constants.REQ_FOLDER +
"?fileName=${header.CamelAwsS3Key}");

the awsS3Client "client" i'm setting in applicationContext.xml as follows

<bean id="defaultAWSCredentialsProviderChain"
class="com.amazonaws.auth.DefaultAWSCredentialsProviderChain" />
<bean id="clientConfiguration" class="com.amazonaws.ClientConfiguration" />

<bean id="client" class="com.amazonaws.services.s3.AmazonS3Client">
	<constructor-arg ref="defaultAWSCredentialsProviderChain" />
	<constructor-arg ref="clientConfiguration" />
</bean>

My issue is the camel is polling and downloading same file again and again.
I want camel to download the files from aws s3 which doesn't exists in
Constants.REQ_FOLDER and refrain camel to download files which already
exists

Can anyone please help me !!

Regards,
Raghav



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-s3-consumer-polling-the-same-file-every-time-tp5784157p5784458.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel s3 consumer polling the same file every time

[jlpedrosa <jl...@gmail.com>]

I think what you are looking for is for an idempotent consumer, there are
many implementations in the camel sources, recently I contributed with one
based on mongodb, but look for your own, there are many.

JL



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-s3-consumer-polling-the-same-file-every-time-tp5784157p5784158.html
Sent from the Camel - Users mailing list archive at Nabble.com.