You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Claus Ibsen (Jira)" <ji...@apache.org> on 2025/07/30 07:04:00 UTC

[jira] [Updated] (CAMEL-22291) camel-aws-s3 - Use nextMarker if file listening is truncated

     [ https://issues.apache.org/jira/browse/CAMEL-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-22291:
--------------------------------
    Summary: camel-aws-s3 - Use nextMarker if file listening is truncated  (was: camel-aws-s3 - Speedup consumer when there is a lot of files)

> camel-aws-s3 - Use nextMarker if file listening is truncated
> ------------------------------------------------------------
>
>                 Key: CAMEL-22291
>                 URL: https://issues.apache.org/jira/browse/CAMEL-22291
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws
>            Reporter: Claus Ibsen
>            Priority: Minor
>             Fix For: 4.14.0
>
>
> If the s3 has more files than maxMessagesPerPoll then s3 returns a truncated list, and on next poll, then you can continue from previous using nextMarker. However nextMarker is only returned if you also use prefix. But when you do not use prefix, then s3 javadoc api suggest to just use the last key from the returned list of s3 objects instead.
>  
> {code:java}
> /**
>  * <p>
>  * When the response is truncated (the <code>IsTruncated</code> element value in the response is <code>true</code>),
>  * you can use the key name in this field as the <code>marker</code> parameter in the subsequent request to get the
>  * next set of objects. Amazon S3 lists objects in alphabetical order.
>  * </p>
>  * <note>
>  * <p>
>  * This element is returned only if you have the <code>delimiter</code> request parameter specified. If the response
>  * does not include the <code>NextMarker</code> element and it is truncated, you can use the value of the last
>  * <code>Key</code> element in the response as the <code>marker</code> parameter in the subsequent request to get
>  * the next set of object keys.
>  * </p>
>  * </note>
>  * 
>  * @return When the response is truncated (the <code>IsTruncated</code> element value in the response is
>  *         <code>true</code>), you can use the key name in this field as the <code>marker</code> parameter in the
>  *         subsequent request to get the next set of objects. Amazon S3 lists objects in alphabetical order. </p>
>  *         <note>
>  *         <p>
>  *         This element is returned only if you have the <code>delimiter</code> request parameter specified. If the
>  *         response does not include the <code>NextMarker</code> element and it is truncated, you can use the value
>  *         of the last <code>Key</code> element in the response as the <code>marker</code> parameter in the
>  *         subsequent request to get the next set of object keys.
>  *         </p>
>  */
> public final String nextMarker() {
>     return nextMarker;
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)