You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "shashank (Jira)" <ji...@apache.org> on 2026/09/25 07:44:00 UTC

[jira] [Created] (CAMEL-25016) Remote file consumers (FTP, FTPS, SFTP, SMB, Azure Files): with noop=true or idempotent=true, a file whose download fails is never retried

shashank created CAMEL-25016:
--------------------------------

             Summary: Remote file consumers (FTP, FTPS, SFTP, SMB, Azure Files): with noop=true or idempotent=true, a file whose download fails is never retried
                 Key: CAMEL-25016
                 URL: https://issues.apache.org/jira/browse/CAMEL-25016
             Project: Camel
          Issue Type: Bug
          Components: camel-core
            Reporter: shashank


In {{GenericFileConsumer.processExchange}} ({{GenericFileConsumer.java:469-513}}) a file is retrieved after {{processStrategy.begin(...)}} ({{:426}}) succeeded. When the retrieve fails, {{tryRetrievingFile}} ({{:516-556}}) throws, and the catch block at {{:499-511}} only removes the file from the in-progress repository, calls the exception handler and returns {{true}}. So {{processBatch}} counts the file as started, and {{GenericFileOnCompletion}}, which is only registered after a successful retrieve ({{:482}}), never runs.

With {{idempotent=true}} and the default {{idempotentEager=true}}, which includes every {{noop=true}} endpoint, the idempotent key is added while polling ({{isValidFile}} -> {{notUnique}}, {{:669-689}}). Nothing removes it after the failed retrieve: {{processBatch}} only removes the key of the files for which {{processExchange}} returned {{false}} ({{:256-266}}, added by CAMEL-21947 for the read-lock-not-acquired case). The next polls skip the file because its key is present, so it is never retried, and only a WARN "Cannot retrieve file ..." is logged once. With the default memory repository this lasts until the application is restarted. With a persistent idempotent repository (JDBC, Infinispan, Hazelcast, ...) the file is never consumed again, not even after a restart.

This happens on the remote file components, whose downloads fail on ordinary transient errors: FTP and FTPS (socket timeout, connection reset, {{426 Transfer aborted}}), SFTP, SMB and Azure Files. The local file component is not affected in practice, because {{FileOperations.retrieveFile}} always returns true ({{FileOperations.java:247-251}}).

With {{preMove}}, {{GenericFileRenameProcessStrategy.begin}} binds the pre moved file to the exchange ({{FILE_EXCHANGE_FILE}}), while the eager key was added for the original path, so removing the key of the exchange file would remove the wrong key.

The same catch block also skips {{processStrategy.abort}}, so the exclusive read lock taken by {{begin}} is not released on a failed retrieve. For the built-in read locks of the remote components ({{none}}, {{changed}}, {{rename}}) that has no lasting effect, but a custom {{exclusiveReadLockStrategy}} bean would keep its lock.

*Reproduction*: the real {{FileConsumer}} / {{GenericFileConsumer}} / process strategies, with a {{FileOperations}} subclass (plugged in through the protected {{FileEndpoint.newFileConsumer}}) whose {{retrieveFile}} throws {{GenericFileOperationFailedException("Connection reset")}} once, as an FTP download failure does. {{FileConsumer}} shares {{GenericFileConsumer}} with the remote consumers, so this exercises the same code path. One file, poll delay 100 ms, result after 3 s:
{noformat}
noop=true                         no failure: processed=1 | 1 failure: processed=0 retrieveCalls=1 fileStillInDir=true keyInRepository=true
idempotent=true (default move)    no failure: processed=1 | 1 failure: processed=0 retrieveCalls=1 fileStillInDir=true keyInRepository=true
{noformat}
Expected: the next poll retrieves and processes the file. Actual: the file is never retried.

A TLA+ model of the consumer (list, begin, retrieve, commit/rollback) violates "a listed file is eventually processed" in 4 states (List -> Begin -> RetrieveFail); with the fix below it holds.

*Proposed fix:* handle a failed or ignored retrieve like a failed begin: call {{processStrategy.abort}} (releases the read lock, deletes a partial local work file, releases the retrieved resources), remove the in-progress entry and the eager idempotent key of the original file (by {{absoluteFileName}} or the {{FILE_IDEMPOTENT_KEY}} snapshot, as {{GenericFileOnCompletion.processStrategyRollback}} does, so it also works with {{preMove}}), and return {{false}}, then report the exception as today.

Behaviour change: a poll in which no file could be downloaded now counts as idle ({{backoffIdleThreshold}}, {{sendEmptyMessageWhenIdle}}, {{greedy}}), the same as a poll in which no read lock was acquired.

_Filed with Claude Code on behalf of allthingssecurity._




--
This message was sent by Atlassian Jira
(v8.20.10#820010)