[jira] [Updated] (CAMEL-25681) camel-support - FileIdempotentRepository.add keeps a key in its cache when the file store cannot be read or written, so the retry of that message is dropped as a duplicate

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25681:
--------------------------------
    Fix Version/s: 4.24.0

> camel-support - FileIdempotentRepository.add keeps a key in its cache when the file store cannot be read or written, so the retry of that message is dropped as a duplicate
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25681
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25681
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> {{FileIdempotentRepository.add(key)}} (main cd2ce899764e, lines 145-176) puts the key into its 1st-level cache first, then checks the file store ({{containsStore}}) and appends the key to it ({{appendToStore}}). When the store cannot be read or written (disk full, permissions, an I/O error of a network file system) these methods throw a {{RuntimeCamelException}}, {{add}} throws, and the message fails - but the key stays in the cache. Every later {{add}} of the same key returns {{false}} from the cache, until the application is restarted (a key that was never stored is then gone from the cache too).
> Consequence: the redelivery of the failed message is taken for a duplicate and skipped, also after the disk has space again. For example, an {{idempotentConsumer}} with a {{FileIdempotentRepository}} after a JMS consumer: during a "disk full" period every message fails (logged), the broker redelivers them, and the redeliveries are filtered as duplicates and acknowledged - lost without a further trace. (The file consumer with {{idempotentRepository}} calls {{add}} while it polls; what it does with the file after such a failure was not checked.)
> {{KafkaIdempotentRepository.add}} (camel-kafka, lines 352-363 on main) already removes the key from its cache again when storing it (broadcasting it on the topic) fails. The other repositories of the project have one store only.
> h3. Reproduction
> New {{FileIdempotentRepositoryFailedStoreTest}} (camel-support):
> * {{appendToStore}} fails once ("No space left on device"), then works: {{[the retry of a key that was not stored] Expecting value to be true but was false}};
> * the store cannot be read (a directory took its place), then it can: the same failure.
> * A third test (passes on main too) pins that a key found in the store is still put into the cache.
> h3. Proposed fix
> Put the key into the cache only after it was found in the store or appended to it. The order of the store accesses does not change, and a key found in the store is still cached (CAMEL-12148). No option or behaviour change for a store that works, so no upgrade guide entry.
> Tests with the fix: camel-support 136 run, 0 failures; camel-core tests matching {{*Idempotent*}}: 77 run, 0 failures. Not run: the full camel-core suite.
> Not changed: when {{trunkStore}} (the maximum file size is reached) fails after the key was appended, {{add}} still throws although the key is stored, so the retry is a duplicate; and {{remove}} removes the key from the cache before the store, so a failing store keeps it in the file (but {{contains}} reads the file and still answers {{true}}).
> How it was found: a review of the repositories that keep a cache in front of a store (backlog item of an earlier sweep); the test confirmed it.
> Affected: main (cd2ce899764e; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x put the key into the cache before the store accesses the same way (read, not run).
> Duplicate check (2026-10-15): JIRA text "FileIdempotentRepository" (17 issues, the latest CAMEL-24953 / CAMEL-24954 about the idempotent consumer, none about a failing store); GitHub pull requests "FileIdempotentRepository" (#22114, camel-atom; nothing related).
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)