You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Yap Poh Soon (JIRA)" <ji...@apache.org> on 2015/03/03 21:45:05 UTC
[jira] [Updated] (CAMEL-8431) Consume all files in aws S3 bucket
where deleteAfterRead = false
[ https://issues.apache.org/jira/browse/CAMEL-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yap Poh Soon updated CAMEL-8431:
--------------------------------
Description:
The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false. The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().
After some code chasing, i think, the root cause is because the ListObjectsRequest does not has the previous marker value. Hence, i wonder if we could do something like below:
{quote}
{color:blue}private string marker; // new line to define the marker string {color}
..
..
..
ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
listObjectsRequest.setBucketName(bucketName);
listObjectsRequest.setPrefix(getConfiguration().getPrefix());
listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
{color:blue}listObjectsRequest.setMarker(marker); // new line to set the marker {color}
ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
{color:blue}marker = listObjects.getMarker(); // where marker is track
{color}{quote}
was:
The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false. The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().
After some code chasing, i discover the root cause if because the ListObjectsRequest does not has the previous marker value. Hence, i wonder if we could do something like below:
{quote}
{color:blue}private string marker; // new line to define the marker string {color}
..
..
..
ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
listObjectsRequest.setBucketName(bucketName);
listObjectsRequest.setPrefix(getConfiguration().getPrefix());
listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
{color:blue}listObjectsRequest.setMarker(marker); // new line to set the marker {color}
ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
{color:blue}marker = listObjects.getMarker(); // where marker is track
{color}{quote}
> Consume all files in aws S3 bucket where deleteAfterRead = false
> ----------------------------------------------------------------
>
> Key: CAMEL-8431
> URL: https://issues.apache.org/jira/browse/CAMEL-8431
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Affects Versions: 2.14.0
> Reporter: Yap Poh Soon
>
> The current AWS S3Consumer class has a problem if user supply the parameter deleteAfterRead=false. The S3Consumer will always consume the same files over and over again with the size of getMaxMessagesPerPoll().
> After some code chasing, i think, the root cause is because the ListObjectsRequest does not has the previous marker value. Hence, i wonder if we could do something like below:
> {quote}
> {color:blue}private string marker; // new line to define the marker string {color}
> ..
> ..
> ..
> ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
> listObjectsRequest.setBucketName(bucketName);
> listObjectsRequest.setPrefix(getConfiguration().getPrefix());
> listObjectsRequest.setMaxKeys(maxMessagesPerPoll);
> {color:blue}listObjectsRequest.setMarker(marker); // new line to set the marker {color}
> ObjectListing listObjects = getAmazonS3Client().listObjects(listObjectsRequest);
> {color:blue}marker = listObjects.getMarker(); // where marker is track
> {color}{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)