[jira] [Commented] (CAMEL-24090) camel-file: eagerly-added idempotent keys are not drained when pollDirectory throws mid-poll - scanned files are never consumed (List drain overload missed by CAMEL-21830)

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096671#comment-18096671 ] 

Claus Ibsen commented on CAMEL-24090:
-------------------------------------

Fixed via PR https://github.com/apache/camel/pull/24745 (merged to main).

> camel-file: eagerly-added idempotent keys are not drained when pollDirectory throws mid-poll - scanned files are never consumed (List drain overload missed by CAMEL-21830)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24090
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24090
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-file, camel-ftp
>    Affects Versions: 4.21.0
>            Reporter: Federico Mariani
>            Assignee: Claus Ibsen
>            Priority: Major
>              Labels: code-review
>             Fix For: 4.22.0
>
>         Attachments: FileConsumerIdempotentEagerKeyLeakOnPollErrorIssueTest.java
>
>
> h3. Problem
> With {{idempotent=true}} (implied by {{noop=true}}) and default {{idempotentEager=true}}, keys are added to the idempotent repository during directory scanning ({{isValidFile}} -> {{notUnique}}). If {{pollDirectory}} then throws (FTP disconnect mid-scan, I/O error in a subdirectory, filter expression error), the catch in {{GenericFileConsumer.poll()}} (GenericFileConsumer.java:146-157) drains via the *List* overload of {{removeExcessiveInProgressFiles}} (line 310-315), which only removes in-progress keys - the eagerly-added idempotent keys stay behind:
> {code:java}
> protected void removeExcessiveInProgressFiles(List<GenericFile<T>> files) {
>     for (GenericFile file : files) {
>         String key = file.getAbsoluteFilePath();
>         endpoint.getInProgressRepository().remove(key);
>         // no removeExcessiveIdempotentFile(...), unlike the Deque overload above it
>     }
> }
> {code}
> Every subsequent poll sees {{add(key) == false}} -> the never-processed files are treated as already consumed and silently skipped forever (until 1000-entry LRU eviction/restart for the memory repo; permanently for persistent repos). Affects file, FTP, FTPS and SFTP consumers - remote consumers are the likeliest trigger (mid-scan disconnect).
> h3. History
> CAMEL-21830 (130bde60462b, Mar 2025) fixed exactly this leak in the *Deque* overload used for batch draining, adding {{removeExcessiveIdempotentFile(file, null)}} there - the List overload used on the poll-exception path was overlooked. The asymmetry between the two drain methods shows this is an oversight, not intent.
> h3. Suggested fix
> Mirror the Deque overload: in the List drain, when {{isIdempotentEager()}} and an idempotent repository is configured, call {{removeExcessiveIdempotentFile(file, null)}}. Consider extracting a shared drain helper so a third incarnation cannot appear.
> h3. Reproducer
> Attached failing JUnit test (place in {{core/camel-core/src/test/java/org/apache/camel/component/file/}}, where the file component tests live). A filter throws on the second file during the first poll only; the first file's eagerly-added key leaks and the file is never consumed on subsequent polls.
> ----
> _This issue was researched and filed by Claude Code on behalf of [~croway] (GitHub: Croway), as part of a deep code review of camel-file and camel-ftp. A failing JUnit reproducer is attached; it fails deterministically on current main (4.22.0-SNAPSHOT)._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)