You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2025/04/28 21:09:00 UTC
[jira] [Assigned] (ARTEMIS-5446) Memory leak on Artemis backup node with failover & failback
[ https://issues.apache.org/jira/browse/ARTEMIS-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Bertram reassigned ARTEMIS-5446:
---------------------------------------
Assignee: Justin Bertram
> Memory leak on Artemis backup node with failover & failback
> -----------------------------------------------------------
>
> Key: ARTEMIS-5446
> URL: https://issues.apache.org/jira/browse/ARTEMIS-5446
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.36.0
> Reporter: Jean-Pascal Briquet
> Assignee: Justin Bertram
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2025-04-24-11-16-45-740.png, image-2025-04-24-11-20-25-598.png, image-2025-04-24-11-24-45-343.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Description:*
> Backup nodes may encounter OOM errors when the primary become unavailable and the backup transitions to live. This issue impacts HA, as the backup node may become unresponsive and block until the node is restarted by an operator.
> *Analysis:*
> Upon analyzing a heap dump of a backup node, it appears that instances of object of type {{PostOfficeImpl}} accumulate on the heap each time the primary node is restarted. These {{PostOfficeImpl}} objects (and related objects like {{QueueImpl}}, {{DivertImpl}}, {{ClusterConnectionImpl}}, ...) are not removed by the GC.
> In large configuration (1500 queues or more), it can fill up the heap memory quickly. I tried to traceback the source of the problem, but it goes a bit far into Artemis internals.
> Once this state is reached, OOM errors happens randomly in various stack traces:
> {code:java}
> Caused by: java.lang.OutOfMemoryError: Java heap space{code}
> *Reproduction Scenario:*
> * Start a primary/backup pair.
> * Primary node is active and the backup node is synchronized
> * Capture a JVM heap dump (at this stage only one single {{PostOfficeImpl}} instance exists on the heap)
> Repeat the following steps multiple times:
> * Stop the primary node
> * Wait for the backup to activate
> * Start the primary node to trigger failback
> * Wait for the primary to activate
> After several cycles perform a last JVM heap dump. You will observe multiple {{PostOfficeImpl}} instances lingering in the heap.
> *Example:*
> !image-2025-04-24-11-16-45-740.png|width=503,height=175!
> Another example with a high number of Queues in configuration (see retained heap size).
> !image-2025-04-24-11-20-25-598.png|width=749,height=214!
> Each instance of {{PostOfficeIml}} has a retained size of 970MB
> !image-2025-04-24-11-24-45-343.png|width=557,height=256!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact