[jira] [Created] (ARTEMIS-5446) Memory leak on Artemis backup node with HA replication policy and Zookeeper quorum

["Jean-Pascal Briquet (Jira)" <ji...@apache.org>]

Jean-Pascal Briquet created ARTEMIS-5446:
--------------------------------------------

             Summary: Memory leak on Artemis backup node with HA replication policy and Zookeeper quorum
                 Key: ARTEMIS-5446
                 URL: https://issues.apache.org/jira/browse/ARTEMIS-5446
             Project: ActiveMQ Artemis
          Issue Type: Bug
          Components: Broker
    Affects Versions: 2.36.0
            Reporter: Jean-Pascal Briquet
         Attachments: image-2025-04-24-11-16-45-740.png, image-2025-04-24-11-20-25-598.png, image-2025-04-24-11-24-45-343.png

*Description:*
Backup nodes may encounter OOM errors when the primary become unavailable and the backup transitions to live.
This issue impacts HA, as the backup node may become unresponsive and block until the node is restarted by an operator.

 

*Analysis:*

Upon analyzing a heap dump of a backup node, it appears that instances of object of type PostOfficeImpl accumulate on the heap each time the primary node is restarted.
These PostOfficeImpl objects (and related objects like QueueImpl, DivertImpl, ClusterConnectionImpl, ...) are not removed by the GC.

In large configuration (1500 queues or more), it can fill up the heap memory quickly.
I tried to traceback the source of the problem, but it goes a bit far into Artemis internals.

Once this state is reached, OOM errors happens randomly in various stack traces:
{code:java}
Caused by: java.lang.OutOfMemoryError: Java heap space{code}
 

*Reproduction Scenario:*
 * Start a primary/backup pair.
 * Primary node is live and the backup node is synchronized
 * Capture a JVM heap dump (at this stage only one single PostOfficeImpl instance exists on the heap)

Repeat the following steps multiple times:
 * Stop the primary node
 * Wait for the backup to become live
 * Start the primary node
 * The backup give back the lead to the primary
 * Wait for the primary to become live

After several cycles, perform a last JVM heap dump. You will observe multiple PostOfficeImpl instances lingering in the heap.


*Example:*

!image-2025-04-24-11-16-45-740.png|width=503,height=175!

 

Another example with a high number of Queues in configuration (see retained heap size).

!image-2025-04-24-11-20-25-598.png|width=749,height=214!

Each instance of PostOfficeIml has a retained size of 970MB

!image-2025-04-24-11-24-45-343.png|width=557,height=256!

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact