[jira] [Work started] (CAMEL-23686) Reduce Exchange memory pressure and fix pooled exchange issues

["Guillaume Nodet (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on CAMEL-23686 started by Guillaume Nodet.
-----------------------------------------------
> Reduce Exchange memory pressure and fix pooled exchange issues
> --------------------------------------------------------------
>
>                 Key: CAMEL-23686
>                 URL: https://issues.apache.org/jira/browse/CAMEL-23686
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-core
>            Reporter: Guillaume Nodet
>            Assignee: Guillaume Nodet
>            Priority: Major
>              Labels: memory, performance
>
> h2. Context
> Profiling Camel routes under high throughput (timer + splitter producing ~1M msg/s) reveals several memory and allocation inefficiencies in the Exchange lifecycle. This issue tracks concrete improvements identified through heap histogram analysis and JFR profiling.
> h2. Findings
> h3. 1. Dev profile forces messageHistory=true, overriding user properties (CRITICAL)
> {{ProfileConfigurer.configureCommon()}} (line 102) unconditionally sets {{messageHistory=true}} in dev mode. This overrides any user setting of {{camel.main.messageHistory=false}} in application.properties, since the profile configurer runs after property loading.
> *Impact:* With pooled exchanges in dev mode, {{DefaultMessageHistory}} instances accumulate unbounded — in our test, 2.95M instances consumed 236MB, ballooning heap from 75MB to 696MB. The history list grows because pooled exchanges recycle the exchange object but the debugger re-creates message history entries on each reuse.
> h3. 2. Exchange pooling only covers consumer exchanges (~40%)
> The {{PooledExchangeFactory}} only provides pooled exchanges for the consumer's initial exchange. Sub-exchanges created by Splitter, Multicast, and RecipientList use regular {{DefaultExchange}} instances.
> In a pipeline route with a splitter, 524K out of 1.23M exchanges were pooled (42%). The remaining 703K (58%) were regular {{DefaultExchange}} instances, bypassing the pool entirely.
> h3. 3. Per-exchange allocation is ~600 bytes across 10-12 objects
> Each exchange allocates:
> ||Object||Bytes||
> |DefaultExchange / DefaultPooledExchange|64-80|
> |ExtendedExchangeExtension|80|
> |EnumMap x2 (properties + internal)|80|
> |DefaultMessage|48|
> |CopyOnWriteHeadersMap|24|
> |CaseInsensitiveMap|48|
> |DefaultUnitOfWork|56|
> |ReentrantLock + NonfairSync|48|
> |ConcurrentLinkedDeque (routes)|24|
> |*Total*|*~552 bytes*|
> At 1M exchanges/s, this generates ~552MB/s allocation rate just for exchange infrastructure.
> h3. 4. UnitOfWork is overweight for common single-route exchanges
> * {{ConcurrentLinkedDeque<Route> routes}} — eagerly allocated, but typically holds only 1 entry. A simple field with lazy upgrade to deque would save allocation.
> * {{ReentrantLock}} — allocated per UoW even though most exchanges are single-threaded. Could be lazily created only when threading is detected.
> h3. 5. ExtendedExchangeExtension always allocated
> {{ExtendedExchangeExtension}} (80 bytes) is created for every exchange, even though most exchanges never use extended features. Lazy initialization would save 80 bytes per exchange.
> h2. Test Environment
> * JDK: Temurin 21.0.9
> * Camel: 4.21.0-SNAPSHOT (with PR #23738 CoW headers + PR #23766 O(1) CaseInsensitiveMap)
> * Route: Timer(period=1) -> Split(1000 tokens) -> Direct -> CBR -> Direct -> mock
> * Duration: 60s, heap histogram captured at 35s
> h2. Benchmark Results
> ||Route||Profile||Heap Used||Metaspace||Threads||
> |Baseline|prod + pooled|75 MB|42 MB|34|
> |Baseline|dev + pooled|696 MB|43 MB|35|
> |Pipeline|dev default|3,194 MB|45 MB|35|
> |Pipeline|prod + pooled|1,270 MB|45 MB|34|
> |HTTP|prod + pooled|95 MB|56 MB|58|
> h2. Positive findings
> * PR #23738 (CopyOnWriteHeadersMap) is working correctly — header copies are avoided
> * PR #23766 (O(1) CaseInsensitiveMap) is active — TreeMap entries seen in histograms are from JMX infrastructure, not headers
> * HTTP component adds a fixed 24 threads + 14MB metaspace (per-component, not per-exchange)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)