[jira] [Resolved] (CAMEL-25607) camel-mongodb-gridfs - with the default TimeStamp query strategy (and PersistentTimestamp) the consumer routes every file uploaded since it started again at each poll, every 500 ms

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-25607.
---------------------------------
    Resolution: Fixed

> camel-mongodb-gridfs - with the default TimeStamp query strategy (and PersistentTimestamp) the consumer routes every file uploaded since it started again at each poll, every 500 ms
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25607
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25607
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-mongodb-gridfs
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> {{GridFsConsumer}} with a timestamp query strategy ({{queryStrategy=TimeStamp}}, the default, and {{PersistentTimestamp}}) asks for the files whose {{uploadDate}} is greater than the date of the newest file it has routed. Since CAMEL-17121 (3.13, the loop was replaced by a repeated task) that date is lost between the polls ({{GridFsConsumer}} at main a7169d7c6ab6):
> {code:java}
> // run(), lines 128-133: the start date is captured once
> Date finalFromDate = fromDate;
> task.run(endpoint.getCamelContext(),
>         () -> processCollection(finalFromDate, usesTimestamp, persistsTimestamp, usesAttribute, finalPtsCollection,
>                 finalPersistentTimestamp));
> // processCollection(Date fromDate, ...), lines 180-183: moves the parameter of this call only
> if (usesTimestamp && file.getUploadDate().compareTo(fromDate) > 0) {
>     fromDate = file.getUploadDate();
>     dateModified = true;
> }
> {code}
> Every poll therefore queries {{uploadDate > <date the consumer started>}} again. A file uploaded after the consumer started is routed by the first poll that sees it and then again by every following poll, every {{delay}} (500 ms by default), for as long as the route runs. Before CAMEL-17121 {{fromDate}} was a local variable of the loop and the file was routed once.
> Configuration: the defaults. {{from("mongodb-gridfs:myDb?database=test")}} and one file created in the bucket are enough; the route receives that file about twice per second.
> With {{PersistentTimestamp}} the effect is the same until the route is restarted (the stored timestamp is written again at every poll, with the same value; a restart then reads it and starts after the file). The strategies that use the file attribute ({{FileAttribute}}, {{TimeStampAndFileAttribute}}, {{PersistentTimestampAndFileAttribute}}) are not affected: their query also asks for files without the attribute.
> The integration test of the module ({{GridFsConsumerIT}}) does not see it: all files have the same body, and the mock endpoint is asserted as soon as the expected number of messages has arrived, so a file that is routed again a poll later is either not seen or looks like the next expected message. CAMEL-24905 (flaky {{GridFsConsumerIT.testAttribute}}, "Expected: <1> but was: <2>") is possibly related, not reproduced: the test that failed there uses the {{FileAttribute}} strategy, which is not affected, and the root causes found in that ticket were the shared mock endpoint and buckets that were not cleaned between the tests.
> h3. Reproduction
> New {{GridFsConsumerTimestampTest}} (3 tests, no MongoDB needed): the real {{GridFsConsumer}} polls a fake bucket (Mockito) that answers the {{uploadDate > date}} query of each poll from a list of files, the way the server does. All three fail with the consumer of main:
> * {{timestampStrategyRoutesAFileOnce}}: {{three polls saw the file, it must be routed by the first of them only ==> expected: <[first]> but was: <[first, first, first, first]>}} (three or four times, depending on the poll that is running when the test asserts)
> * {{timestampStrategyRoutesALaterFileOnce}}: {{expected: <[first, second]> but was: <[first, first, first, first, second, first, second, first, second]>}}
> * {{persistentTimestampStrategyRoutesAFileOnce}}: {{expected: <[first]> but was: <[first, first, first]>}}
> h3. Proposed fix
> Keep the date in a field of the consumer, so that a poll starts from the date the previous poll reached. The field is {{volatile}}: {{run()}} sets the start value on the thread of the consumer's {{executor}} before it schedules the polls, and the polls read and move it on the thread of the {{taskExecutor}}. Nothing else changes.
> A stop and start of the consumer is as before: {{doStart}} runs {{run()}} again, which sets the date to "now" for {{TimeStamp}} and reads the stored timestamp for {{PersistentTimestamp}}.
> Behaviour change, with a 4.24 upgrade-guide entry: a file is routed once.
> Not changed, seen in the same method: a failed exchange still moves the timestamp (and sets the file attribute to {{done}}), an exception thrown by {{process}} is still swallowed, and an exception of the poll itself (MongoDB not reachable) still ends the repeated task. These were the same before CAMEL-17121 or are a different defect, and would be separate tickets.
> camel-mongodb-gridfs tests with the fix: 3 run, 0 failures (the module has no other unit test) (the integration tests need a MongoDB container and were not run).
> How it was found: a small TLA+ model of the polls (upload, query, route, end of poll). For the code of main TLC reports a violation of "a file is routed at most once" after two polls (7 states) and of "the timestamp is stored only when it moved"; with the date kept between the polls both hold, and no file is skipped. The model is a sketch of the code; the evidence is the test above.
> Affected: main (a7169d7c6ab6, tested). Read, not run: camel-4.23.x (197e70656a41), camel-4.22.x (8dfae2fa437c), camel-4.18.x (c3fd5317a5a0), camel-4.14.x, camel-3.14.x and the camel-3.13.0 tag have the same lines ({{finalFromDate}} captured by the task, {{processCollection(Date fromDate, ...)}}), so every release since 3.13.0. The camel-3.12.0 tag still has the loop with a local variable.
> Duplicate check (2026-10-13): JIRA text "GridFsConsumer" (3: CAMEL-24286, CAMEL-24272, CAMEL-17121, none about duplicates), "gridfs consumer" (3), "gridfs duplicate" (2), "gridfs timestamp" (1), "mongodb-gridfs" (26, none about the consumer routing a file more than once); GitHub pull requests "gridfs": only #26787 (CAMEL-24905, test only) touches the consumer tests.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)