[jira] [Created] (CAMEL-20835) OOM using RecipientList

["Arthur Naseef (Jira)" <ji...@apache.org>]

Arthur Naseef created CAMEL-20835:
-------------------------------------

             Summary: OOM using RecipientList
                 Key: CAMEL-20835
                 URL: https://issues.apache.org/jira/browse/CAMEL-20835
             Project: Camel
          Issue Type: Bug
          Components: camel-core
    Affects Versions: 4.6.0, 3.10.0
            Reporter: Arthur Naseef


Out Of Memory (OOM) occurs when using the Recipient List with a large number of dynamic URLs.  For example:
 
    .recipientList(simple("http://\{{downstream-server}}/employee/${header.emplId}"))
 
with a large number of values for ${header.emplId} leads to the OOM.
 
REPRODUCER:
=============
[https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
 
- See the README.md for instructions to reproduce and detect the problem
 
DETAILS
=======
 The MulticastProcessor, which RecipientListProcessor extends, has the following "unlimited" cache:
 
    private final ConcurrentMap<Processor, Processor> errorHandlers = new ConcurrentHashMap<>();
 
Entries are added to this map for every unique processor created - every unique URL generates a unique processor.  The entries themselves are wrapped processor instances for error handling IIUC (to support the custom error handling used by multicast and recipient-list).  Entries are only removed from this map on shutdown.  Ironically, there is an LRUCache for the processors themselves, with a default maximum size of 1000, so the wrapped processors may get recreated even though the error handler remains in the map indefinitely.
 
*IMPACT VERSIONS:*
*================*
Appears to impact versions >= 3.10.0
*COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
*=======*
It appears this commit introduced the use of the errorHandlers "unlimited" cache for recipient lists.
 
*FOLLOW-UP*
*==========*
I have ideas and questions for implemeting a fix:
    - IDEA 1: We can use an LRUCache for this data structure as well.
    - Does it make more sense to remove the entries from errorHandlers when the related Processor entry is removed from it's LRUCache?
    - IDEA 2: setting on recipient list to disable the errorHandler cache (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)