You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "shashank (Jira)" <ji...@apache.org> on 2026/10/20 05:42:00 UTC

[jira] [Work started] (CAMEL-25856) camel-file - the per-file lock of the file producer is evicted after 100 other file names, so two exchanges can append to the same file at the same time and mix its content

     [ https://issues.apache.org/jira/browse/CAMEL-25856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on CAMEL-25856 started by shashank.
----------------------------------------
> camel-file - the per-file lock of the file producer is evicted after 100 other file names, so two exchanges can append to the same file at the same time and mix its content
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25856
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25856
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-file
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>
> {{GenericFileProducer}} (main 20cdb309f7d0) takes a lock per target file name around the write, so that two exchanges do not write the same file at the same time (added by CAMEL-3584 for concurrent appends). The locks are kept in an LRU cache of 100 entries:
> {code:java}
> // assume writing to 100 different files concurrently at most for the same
> // file producer
> private final Map<String, Lock> locks = LRUCacheFactory.newLRUCache(100);
> ...
> Lock lock = locks.computeIfAbsent(target, f -> new ReentrantLock());
> lock.lock();
> try {
>     processExchange(exchange, target);
> } finally {
>     // do not remove as the locks cache has an upper bound
>     lock.unlock();
> {code}
> The cache does not count the files being written at the same time: it keeps the 100 file names used last. While one exchange is still writing a file (a large body, a stream, a slow disk or share), any 100 other file names written by the same producer evict its lock. The next exchange for the same file then gets a new lock and writes at the same time as the first one. With {{fileExist=Append}} the two contents are mixed in the file; with the default {{fileExist=Override}} two writers truncate and write the same file at the same time. Nothing fails and nothing is logged.
> This needs one producer that is used by more than one thread (threads(), parallel split or multicast, concurrent consumers in front of it) and that writes many file names, such as one file per customer, order or day with {{fileName=...}}. Two threads are enough: one writes the large file, the other writes 100 small files and then the same large file again.
> *Fix*: keep a lock in a {{ConcurrentHashMap}} while an exchange holds or waits for it (a count of users, changed in {{compute}} / {{computeIfPresent}}) and remove it when the last one is done. The map then holds only the file names being written, so it stays bounded without evicting a lock in use. camel-smb uses the same producer code; the FTP/SFTP producers do not take this lock.
> *Tests*: new {{FileProducerSameFileLockTest}} in camel-file (2 tests, no route: the file producer is called from two threads). A first exchange appends {{AAAA}}, then its body stream blocks; meanwhile the producer writes other files; a second exchange appends {{BBBB}} to the same file. With 10 other files the second exchange waits (passes on main). With 150 other files, on main: {{the two appends to same.txt are mixed ==> expected: <AAAAAAAABBBB> but was: <AAAABBBBAAAA>}}. Both pass with the fix. camel-core file component tests ({{org.apache.camel.component.file.**}}) with the fix: 474 run, 0 failures, 0 errors, 12 skipped (manual tests).
> *How it was found*: a review of bounded caches ({{LRUCacheFactory}}) whose entries are in use when they are evicted, then the JUnit test above.
> *Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x have the same LRU cache of 100 locks (read, not run).
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)