You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org> on 2026/03/28 22:29:44 UTC

[PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

thomas-gantenbein-tga opened a new pull request, #22311:
URL: https://github.com/apache/camel/pull/22311

   # Description
   To reduce cpu load when many files are processed, let's use a LinkedHashMap instead of SimpleLRUCache as the default inProgressRepository for the file component. See https://issues.apache.org/jira/browse/CAMEL-23267 and [#camel > file consumer does not release items in SimpleLRUCache](https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/file.20consumer.20does.20not.20release.20items.20in.20SimpleLRUCache/with/581658577)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "Croway (via GitHub)" <gi...@apache.org>.
Croway commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4168753774

   Hi @thomas-gantenbein-tga `I could also use this factory method in IBMCOSEndpoint and AWS2S3Endpoint` this would be nice


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus closed pull request #22311: CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com…
URL: https://github.com/apache/camel/pull/22311


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4150604377

   :star2: Thank you for your contribution to the Apache Camel project! :star2:
     :robot: CI automation will test this PR automatically.
   
     :camel: Apache Camel Committers, please review the following items:
   
     * First-time contributors **require MANUAL approval** for the GitHub Actions to run
     * You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
     * You can label PRs using `build-all`, `build-dependents`, `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
     * Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
     
     :warning: Be careful when sharing logs. Review their contents before sharing them publicly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4212847331

   @thomas-gantenbein-tga we have merged a fix from @essobedo to the LRU cache, so you are welcome to test with latest source code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4165128592

   If you want me to, I could also use this factory method in `IBMCOSEndpoint` and `AWS2S3Endpoint`. Not sure, but looks like they would have the same problem as the GenericFileEndpoint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4222060435

   RemoteFileEndpoint
   SmbEndpoint
   IBMCOSEndpoint
   FileEndpoint
   GenericFileEndpoint
   AWS2S3Endpoint


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4209263084

   > Hi @thomas-gantenbein-tga `I could also use this factory method in IBMCOSEndpoint and AWS2S3Endpoint` this would be nice
   
   Okay, I've updated the PR accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4222441259

   @thomas-gantenbein-tga do you mind changing the `MemoryIdempotentRepository` in the core module to just use the LinkedHashMap code that you have, and remove the LRUCache then it works generic for all.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4207148481

   I agree with @gnodet that the LRUCache is to be improved/fixed. We use this elsewhere in Camel. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4222324071

   `LinkedHashMap` is not concurrent, but it doesn't need to be in this context — all access in `MemoryIdempotentRepository` is already guarded by a `ReentrantLock` (`cacheAndStoreLock`). Every `add()`, `contains()`, `remove()`, and `clear()` acquires the lock before touching the map. So the underlying map only needs to be a plain `Map`, not a concurrent one.
   
   The current `SimpleLRUCache` with its internal `ConcurrentHashMap` is actually overkill here — you get double-locking (repository lock + CHM segment locks) for no benefit.
   
   Caffeine would be the right choice if we needed a standalone concurrent LRU cache without external synchronization, but for the in-progress repository pattern (add, process, remove — no LRU access-order reordering needed), a locked `LinkedHashMap` with `removeEldestEntry` is the simplest correct solution.
   
   I agree with @davsclaus's suggestion to just make `MemoryIdempotentRepository` use `LinkedHashMap` by default.
   
   _Claude Code on behalf of Guillaume Nodet_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4165241839

   About "Fixing the SimpleLRUCache": Yeah, that was one of my initial thoughts as well. But then maybe there's a reason for the current implementation. Maybe those ghost entries are only a problem when the SimpleLRUCache is used as the store for a inProgressRepository?
   
   So instead of fixing something that is not obviously broken, I'd rather fix what I know _is_ broken, and keep the rest for another ticket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4222039348

   <!-- ci-tested-modules -->
   :test_tube: **CI tested the following changed modules:**
   
   - `components/camel-aws/camel-aws2-s3`
   - `components/camel-file`
   - `components/camel-ibm/camel-ibm-cos`
   - `core/camel-support`
   
   > :information_source: Dependent modules were not tested because the total number of affected modules exceeded the threshold (50). Use the `test-dependents` label to force testing all dependents.
   
   <details><summary>Build reactor — dependencies compiled but only changed modules were tested (4 modules)</summary>
   
   - Camel :: AWS2 S3
   - Camel :: File
   - Camel :: IBM :: Cloud Object Storage
   - Camel :: Support
   
   </details>
   
   ---
   :gear: [View full build and test results](https://github.com/apache/camel/actions/runs/24155431659)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4209787653

   > @essobedo maybe you have some comments to this, see the AI analysis from @gnodet 
   
   Hi, I will have a look tomorrow to see if I can find a fix for this use case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4148963092

   _Claude Code on behalf of Guillaume Nodet_
   
   ## Follow-up: What about fixing `SimpleLRUCache` directly?
   
   ### The root cause
   
   `SimpleLRUCache.remove(Object key)` (line 183) does `delegate.remove(key)` but **doesn't decrement `totalChanges`** and **doesn't remove the key's entry from `lastChanges`**. So the deque accumulates "ghost" entries — keys that were added and then removed from the delegate but still sit in the deque.
   
   When `totalChanges` exceeds `2 * maximumCacheSize` (100,000 for the file component), `compressChangesIfNeeded()` fires on every subsequent `put()`. It iterates the entire deque, deduplicates by key, but ghost entries for removed keys still survive deduplication (they're unique keys). The deque never shrinks below the number of distinct ghost keys.
   
   ### What a fix would look like
   
   1. Decrement `totalChanges` in `remove()` to prevent unbounded growth:
   
   ```java
   @Override
   public V remove(Object key) {
       ValueHolder<V> removed = delegate.remove(key);
       if (removed != null) {
           totalChanges.decrementAndGet();
       }
       return extractValue(removed);
   }
   ```
   
   2. Filter out ghost entries in `compressChangesIfNeeded()`:
   
   ```java
   private void compressChangesIfNeeded() {
       if (isQueueFull()) {
           Deque<Entry<K, ValueHolder<V>>> newChanges = new ConcurrentLinkedDeque<>();
           Deque<Entry<K, ValueHolder<V>>> currentChanges = lastChanges.getAndSet(newChanges);
           Set<K> keys = new HashSet<>();
           Entry<K, ValueHolder<V>> entry;
           while ((entry = currentChanges.pollLast()) != null) {
               // skip ghost entries for keys no longer in the cache
               if (keys.add(entry.getKey()) && delegate.containsKey(entry.getKey())) {
                   newChanges.addFirst(entry);
               }
           }
           totalChanges.set(newChanges.size());
       }
   }
   ```
   
   ### Benefits vs. drawbacks
   
   | | Fix `SimpleLRUCache` | This PR's `LinkedHashMap` approach |
   |---|---|---|
   | **Scope** | Fixes all callers (file, AWS S3, IBM COS, any future use) | Only fixes the file component |
   | **Complexity** | Small, targeted change in existing code | Adds a new factory method + new code path |
   | **LRU semantics** | Preserves LRU eviction (access-order) | Switches to insertion-order eviction (FIFO) |
   | **Thread safety** | Already thread-safe (`ConcurrentHashMap` + locks) | Relies on `MemoryIdempotentRepository`'s lock; `getCacheSize()`/`doStop()` are unguarded |
   | **Risk** | Higher — `SimpleLRUCache` is used broadly, subtle concurrency bugs possible | Lower — isolated change, only affects file component default |
   | **Performance** | Slightly heavier (deque + ConcurrentHashMap overhead) | Lighter (plain `LinkedHashMap`, no deque) |
   
   ### Recommendation
   
   Ideally do both: fix `SimpleLRUCache` because the ghost entry bug is a real defect that will bite other callers (AWS S3 and IBM COS endpoints have the same in-progress repository pattern). But also keep the `LinkedHashMap` approach for the in-progress repository — it's simpler, lighter, and a better semantic fit since in-progress files don't need LRU reordering (they're added, processed, then removed). The `SimpleLRUCache` fix could be tracked as a separate JIRA ticket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on code in PR #22311:
URL: https://github.com/apache/camel/pull/22311#discussion_r3018024176


##########
core/camel-support/src/main/java/org/apache/camel/support/processor/idempotent/MemoryIdempotentRepository.java:
##########
@@ -77,6 +79,21 @@ public static IdempotentRepository memoryIdempotentRepository(int cacheSize) {
         return answer;
     }
 
+    /**
+     * Creates a new memory based repository using a {@link java.util.LinkedHashMap} as its store, with the given
+     * maximum capacity. When a new entry is added and the store has reached its maximum capacity, the oldest entry is
+     * removed.
+     */
+    public static IdempotentRepository memoryIdempotentRepositoryInsertionOrder(int cacheSize) {
+        LinkedHashMap<String, Object> map = new LinkedHashMap<>() {
+            @Override
+            protected boolean removeEldestEntry(Entry<String, Object> eldest) {
+                return size() > cacheSize;
+            }
+        };

Review Comment:
   Thanks. I guess I am not supposed to change the name of that `cacheSize` to `maxCacheSize`, since that's annotated with `@Metadata`? Or am I? I'd find that clearer. I find it confusing that `getCacheSize()` and `setCacheSize(int cacheSize)` are not targeting the same thing.
   
   I'm also calling `ServiceHelper.startService(answer);`, since that's also what's done in ` public static IdempotentRepository memoryIdempotentRepository(int cacheSize)`, but I don't have a clue what the benefit of that is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4225070760

   Yeah that is actually a good point to keep the old behaviour and a LRU based algorithm there is better as then it ensures that if you attempt to process the same files it can detect those duplcates better


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4221955306

   We can likely just make the MemoryIdempotentRepository use LinkedHashMap by default and remove the LRUCache as its only used for in progress in the Camel components. Then we can update the 4.19 upgrade guide doc to mention this change.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4224933771

   > Created another PR with this included #22547
   > 
   > @thomas-gantenbein-tga we are working on this as we are releasing soon
   
   Thanks, and sorry for not being quite as responsive as you guys. I have just one question: are you sure nobody needs lru semantics when a idempotentrepo is created? When used as inProgress, certainly there's no need for that. But when used because someone sets idempotent=true on a file consumer - maybe lru has some value and switching to fifo could break stuff in some cases? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4212709389

   See https://github.com/apache/camel/pull/22500


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4224551179

   Created another PR with this included
   https://github.com/apache/camel/pull/22547
   
   @thomas-gantenbein-tga we are working on this as we are releasing soon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4209428797

   > I agree with @gnodet that the LRUCache is to be improved/fixed. We use this elsewhere in Camel.
   
   Yes, it's used in a whole lot of places in Camel, that's why I hesitated to "fix" it. There's a lot of stuff going on about performance (CAMEL-21888), concurrency issues (CAMEL-22176), soft caching (CAMEL-20039) and other stuff in there that I don't find as "simple" as the name of `SimpleLRUCache` suggests.
   
   For example: Say I "fix" the SimpleLRUCache by overriding the "remove" method and iterating over the `AtomicReference<Deque<Entry<K, ValueHolder<V>>>> lastChanges` to remove an entry from both the `Map<K, ValueHolder<V>> delegate` and that `lastChanges` Deque: That might well introduce a performance problem at a different place, wouldn't it?
   
   So personally, I kind of agree with the AI analysis part that says "do both": use that LinkedHashMap where "least recently used" semantics adds overhead but no benefits (that I see, at least) and fix the ghost entry problem in the SimpleLRUCache as well.
   
   Let me know if any other comments from my side are welcome or if you'd like to take this into your own hands from now on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4209210133

   @essobedo maybe you have some comments to this, see the AI analysis from @gnodet 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on code in PR #22311:
URL: https://github.com/apache/camel/pull/22311#discussion_r3017982058


##########
core/camel-support/src/test/java/org/apache/camel/support/MemoryIdempotentRepositoryTest.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.support;
+
+import java.io.IOException;
+
+import org.apache.camel.spi.IdempotentRepository;
+import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class MemoryIdempotentRepositoryTest {
+
+    @Test
+    void repositoryEvictsOldestEntryWhenRepositoryIsFull() throws IOException {
+        final int cacheSize = 5;
+        final int entriesNotFittingInRepository = 4;
+        try (IdempotentRepository repository = MemoryIdempotentRepository.memoryIdempotentRepositoryInsertionOrder(
+                cacheSize)) {
+
+            for (int i = 0; i < cacheSize + entriesNotFittingInRepository; i++) {
+                repository.add(String.valueOf(i));
+            }
+
+            for (int i = entriesNotFittingInRepository; i < cacheSize + entriesNotFittingInRepository; i++) {
+                assertTrue(repository.contains(String.valueOf(i)), "Repository should contain entry " + i);
+            }
+            for (int i = 0; i < cacheSize - entriesNotFittingInRepository; i++) {
+                assertFalse(repository.contains(String.valueOf(i)), "Repository should not contain entry " + i);
+            }

Review Comment:
   Thanks! Great, such a small test, and I managed to mess it up 😢 . Fixed in second commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on code in PR #22311:
URL: https://github.com/apache/camel/pull/22311#discussion_r3005423371


##########
core/camel-support/src/test/java/org/apache/camel/support/MemoryIdempotentRepositoryTest.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.support;
+
+import java.io.IOException;
+
+import org.apache.camel.spi.IdempotentRepository;
+import org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+class MemoryIdempotentRepositoryTest {
+
+    @Test
+    void repositoryEvictsOldestEntryWhenRepositoryIsFull() throws IOException {
+        final int cacheSize = 5;
+        final int entriesNotFittingInRepository = 4;
+        try (IdempotentRepository repository = MemoryIdempotentRepository.memoryIdempotentRepositoryInsertionOrder(
+                cacheSize)) {
+
+            for (int i = 0; i < cacheSize + entriesNotFittingInRepository; i++) {
+                repository.add(String.valueOf(i));
+            }
+
+            for (int i = entriesNotFittingInRepository; i < cacheSize + entriesNotFittingInRepository; i++) {
+                assertTrue(repository.contains(String.valueOf(i)), "Repository should contain entry " + i);
+            }
+            for (int i = 0; i < cacheSize - entriesNotFittingInRepository; i++) {
+                assertFalse(repository.contains(String.valueOf(i)), "Repository should not contain entry " + i);
+            }

Review Comment:
   The loop condition `i < cacheSize - entriesNotFittingInRepository` equals `i < 1`, so this only checks that entry "0" was evicted. It should verify all 4 evicted entries (0, 1, 2, 3) are absent.
   
   ```suggestion
               for (int i = 0; i < entriesNotFittingInRepository; i++) {
   ```



##########
core/camel-support/src/main/java/org/apache/camel/support/processor/idempotent/MemoryIdempotentRepository.java:
##########
@@ -77,6 +79,21 @@ public static IdempotentRepository memoryIdempotentRepository(int cacheSize) {
         return answer;
     }
 
+    /**
+     * Creates a new memory based repository using a {@link java.util.LinkedHashMap} as its store, with the given
+     * maximum capacity. When a new entry is added and the store has reached its maximum capacity, the oldest entry is
+     * removed.
+     */
+    public static IdempotentRepository memoryIdempotentRepositoryInsertionOrder(int cacheSize) {
+        LinkedHashMap<String, Object> map = new LinkedHashMap<>() {
+            @Override
+            protected boolean removeEldestEntry(Entry<String, Object> eldest) {
+                return size() > cacheSize;
+            }
+        };

Review Comment:
   This factory method passes the map to the `MemoryIdempotentRepository(Map)` constructor, which does not set the `cacheSize` field. As a result, `getMaxCacheSize()` (a JMX-exposed `@ManagedAttribute`) will return 0.
   
   Consider setting it:
   ```java
   public static IdempotentRepository memoryIdempotentRepositoryInsertionOrder(int cacheSize) {
       LinkedHashMap<String, Object> map = new LinkedHashMap<>() {
           @Override
           protected boolean removeEldestEntry(Entry<String, Object> eldest) {
               return size() > cacheSize;
           }
       };
       MemoryIdempotentRepository answer = new MemoryIdempotentRepository(map);
       answer.setCacheSize(cacheSize);
       return answer;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4221509623

   Cross-posting:
   
   > Thanks for the feedback. Regarding your comment about the last changes that are not updated on remove, it is intended in this "basic" implementation of an LRU cache to have a remove with a time complexity of O(1). IMHO, we should propose a real alternative based on Caffeine [caffeine-lrucache](https://camel.apache.org/components/3.22.x/others/caffeine-lrucache.html) and/or Guava, proposing a non thread-safe alternative based on a LinkedHashMap as it was before sounds more like a workaround than a real solution, but it is only my point of view.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] CAMEL-23267: Use a LinkedHashMap as inProgressRepository for file com… [camel]

Posted by "thomas-gantenbein-tga (via GitHub)" <gi...@apache.org>.
thomas-gantenbein-tga commented on PR #22311:
URL: https://github.com/apache/camel/pull/22311#issuecomment-4217703840

   Thanks, @davsclaus and @essobedo. I tested my reproducer with the change from https://github.com/apache/camel/pull/22500https://github.com/apache/camel/pull/22500 and yes, that change fixes, as expected, the CPU problem that became evident after 100'000 files had been processed.
   
   I guess you are aware of this, but I'll write it anyway: That change fixes the CPU problem, but it still lets the `lastChanges` queue grow to 100'001 items, of which 100'000 are paths to files that have already been processed and are not "inProgress", i.e. not part of the `delegate` HashMap anymore. Only then it's cleaned up. Then it grows again to 100'001 items, then it's cleaned up again and so on. Maybe that's only me, I just don't find that overly pretty.
   
   Thanks for the fix, anyway. And, of course, feel free to close this PR unmerged if you think the LRU fix is good enough and the change here is not worth the lines it adds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org