[jira] [Updated] (CAMEL-25785) camel-aws2-s3 - streaming upload with timestampGroupingEnabled replaces the object of a time window when the window is uploaded again (batchMessageNumber, batchSize, streamingUploadTimeout), so the earlier messages of the window are lost

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25785:
--------------------------------
    Fix Version/s: 4.24.0

> camel-aws2-s3 - streaming upload with timestampGroupingEnabled replaces the object of a time window when the window is uploaded again (batchMessageNumber, batchSize, streamingUploadTimeout), so the earlier messages of the window are lost
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25785
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25785
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-aws2
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> {{AWS2S3StreamUploadProducer.processWithTimestampGrouping}} (main 7aae27315946, lines 336-450) names the object of a time window with {{generateTimestampBasedFileName(fileName, extension, timestampWindow)}}, which depends only on the window: {{grouped-messages_20240101_0800_0800-0805.txt}}. The upload of a window is completed, and the window removed from {{timestampBasedUploads}}, when it reaches {{batchMessageNumber}} messages (default 10) or {{batchSize}} bytes (lines 418-436), and by the {{streamingUploadTimeout}} task (lines 133-170). The next message of the same window then starts a new multipart upload with the *same key*, and when that upload completes S3 replaces the first object with it:
> {code:java}
> if (!timestampBasedUploads.containsKey(timestampWindow)) {
>     UploadState newState = new UploadState();
>     ...
>     newState.dynamicKeyName = generateTimestampBasedFileName(fileName, extension, timestampWindow);
> {code}
> So every message of the earlier uploads of the window is lost (unless the bucket is versioned), and every exchange completed normally with the ETag of an object that is replaced later. The example of the component documentation (aws2-s3-streaming.adoc, "Timestamp Grouping": a timer every 10 seconds, {{timestampWindowSizeMillis=300000}}, default {{batchMessageNumber}}) sends 30 messages per window: the window is uploaded after the 10th, 20th and 30th message, and the bucket keeps only the last 10. The documentation says the feature writes the messages of a window "to the same S3 file"; it says nothing of objects being replaced. The progressive, random and timestamp naming strategies of the non-grouped mode do give each upload its own key.
> *Fix*: the producer counts the completed uploads of each window (under its lock; an access-ordered map bounded to the last 10000 windows), and a later upload of a window gets that count after the time range: {{grouped-messages_20240101_0800_0800-0805-1.txt}}, {{-2}}, ... The first upload of a window keeps its name. Documentation of the naming in aws2-s3-streaming.adoc, and a note in the 4.24 upgrade guide (the key of the second and later objects of a window is new).
> *Tests*: new {{AWS2S3StreamUploadTimestampWindowOverwriteTest}} (mocked {{S3Client}} that keeps completed objects by key, like a bucket). {{everyMessageOfAWindowIsStoredWhenTheWindowIsUploadedMoreThanOnce}} (the documentation example, 30 messages in one window) fails on main: {{each upload of the window must be its own object, but the bucket has [grouped-messages_20240101_0800_0800-0805.txt] ==> expected: <3> but was: <1>}}. {{everyMessageOfAWindowIsStoredWhenTheTimeoutUploadsTheWindowBeforeItEnds}} (the timeout task runs in the middle of a window) fails on main: {{expected: <2> but was: <1>}}. {{theFirstUploadOfAWindowKeepsItsName}} passes on main and with the fix. {{AWS2S3StreamUploadProducerTimestampTest}} is adapted to the new signature of the name method and checks the suffix. camel-aws2-s3 unit tests: 107 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25781 (pattern: a batch window completed by a size or count limit or a timer while the window is still open; what happens to the next item of that window), then the JUnit test.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same naming in {{processWithTimestampGrouping}}; read, not run). Not run against S3.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)