You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Justin Bertram (Jira)" <ji...@apache.org> on 2024/07/23 16:53:00 UTC
[jira] [Comment Edited] (ARTEMIS-4947) Out of Memory on too many connecting/disconnecting MQTT clients
[ https://issues.apache.org/jira/browse/ARTEMIS-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868125#comment-17868125 ]
Justin Bertram edited comment on ARTEMIS-4947 at 7/23/24 4:52 PM:
------------------------------------------------------------------
Can you provide the code you're running to perform the load test?
Also, can you simplify the test to a single node instead of a cluster? It's not clear how connections are being distributed across the nodes of the cluster, which nodes are experiencing OOME, and which node(s) the heap dumps and screenshots are from. At this point using a cluster is just making everything more difficult to understand & diagnose.
was (Author: jbertram):
Can you provide the code you're running to perform the load test?
> Out of Memory on too many connecting/disconnecting MQTT clients
> ---------------------------------------------------------------
>
> Key: ARTEMIS-4947
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4947
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.35.0
> Reporter: Shirsh Kumar
> Priority: Major
> Attachments: Histogram_otherObjects.png, NettyAcceptor_ConnectionsAllowed_2000.png, QueueImpl_Objects.png, broker.xml, dump_oom.7z.001, dump_oom.7z.002
>
>
> I was trying to use Artemis ActiveMQ broker with Kubernetes using Artemis cloud's [activemq-artemis-operator|https://github.com/artemiscloud/activemq-artemis-operator]
>
> *Usecase* (Short Lived Clients): It consists of clients connecting to broker, subscribing to topics and disconnecting after 1 or 2 minutes.
> The cluster uses the [Artemis Kubernetes image 1.0.28 |https://github.com/artemiscloud/activemq-artemis-broker-kubernetes-image]with underlying [Artemis broker version 2.35.0|https://activemq.apache.org/components/artemis/download/release-notes-2.35.0]
>
> Kubernetes Pods config:
> CPU: 1
> Memory: 2 Gi
> Max ConnectionsAllowed: 2000
>
> So, to test the stability of the system a load test was done on cluster with initial 2 pods and scaling allowed till 3 pods.
> I assumed restriction to keep the system stable by setting ConnectionsAllowed to a value of 2000 should work.
> Persistence had to be disabled as I was getting 5000ms timeout while writing session state to disk (Attached broker.xml).
>
> *Load test:* 7000 new client connections per minute (Client ID's = Some Prefix + Epoch)
> (Clients connect & subscribe and then unsubscribe & disconnect after 1 minute).
>
>
> So, on testing the cluster with this setup I was able to see that for initial duration of 30 mins to 1 hr., the broker pods run fine.
>
> After some time, it is observed that *out of memory* in the pods occurs after which they restart.
>
> I have captured heap dump for various stages of the test:
> 1. Initial Heap Dump: 10 clients connect and then disconnect after 1 minute
> Dump size: 63 MB
> 2. Heap Dump at 83 % memory usage after 15-20 minutes
> Dump size: 370 MB
> 3. Heap dump after OOM
> Dump size: 2.1 GB
>
> From my observations on analyzing the heap dumps (do not have deep knowledge of Artemis), I could see the below:
>
> 1. The *QueueImpl has retained HEAP of 600 MB* which is taking up 48 percent bytes.
> (Attached QueueImpl_Objects.png)
>
> 2. There is a *mismatch* between the number of NettyServerConnections object's connectedClients table and the number of session states stored in MQTT
> (Attached NettyAcceptor_ConnectionsAllowed_2000.png & Histogram_otherObjects.png)
>
> 3. The thread overview at the time of dump has one active thread which is trying to removeSubscriptions as below:
>
>
> {code:java}
> Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3dad535f)
> Status --> alive, Runnable
> Other all threads at this point are --> alive, blocked on monitor enter.
> -------------> Thread stack ----------->
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @ 0xb56ff6e0 : Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3dad535f)
> at java.util.concurrent.ConcurrentHashMap.forEach(Ljava/util/function/BiConsumer;)V (ConcurrentHashMap.java:1603)
> at org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.getDirectBindings(Lorg/apache/activemq/artemis/api/core/SimpleString;)Ljava/util/Collection; (SimpleAddressManager.java:165)
> at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.getDirectBindings(Lorg/apache/activemq/artemis/api/core/SimpleString;)Ljava/util/Collection; (PostOfficeImpl.java:1097)
> at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeAddressInfo(Lorg/apache/activemq/artemis/api/core/SimpleString;Z)Lorg/apache/activemq/artemis/core/server/impl/AddressInfo; (PostOfficeImpl.java:883)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.removeAddressInfo(Lorg/apache/activemq/artemis/api/core/SimpleString;Lorg/apache/activemq/artemis/core/security/SecurityAuth;Z)V (ActiveMQServerImpl.java:4015)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.removeAddressInfo(Lorg/apache/activemq/artemis/api/core/SimpleString;Lorg/apache/activemq/artemis/core/security/SecurityAuth;)V (ActiveMQServerImpl.java:3989)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(Lorg/apache/activemq/artemis/api/core/SimpleString;Lorg/apache/activemq/artemis/core/security/SecurityAuth;ZZZZ)V (ActiveMQServerImpl.java:2523)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(Lorg/apache/activemq/artemis/api/core/SimpleString;Lorg/apache/activemq/artemis/core/security/SecurityAuth;ZZZ)V (ActiveMQServerImpl.java:2461)
> at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.deleteQueue(Lorg/apache/activemq/artemis/api/core/SimpleString;Z)V (ServerSessionImpl.java:1212)
> at org.apache.activemq.artemis.core.protocol.mqtt.MQTTSubscriptionManager.removeSubscriptions(Ljava/util/List;Z)[S (MQTTSubscriptionManager.java:271)
> at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handleUnsubscribe(Lio/netty/handler/codec/mqtt/MqttUnsubscribeMessage;)V (MQTTProtocolHandler.java:390)
> at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.act(Lio/netty/handler/codec/mqtt/MqttMessage;)V (MQTTProtocolHandler.java:180)
> at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler$$Lambda$1042+0x00007f62588397b0.onMessage(Ljava/lang/Object;)V ()
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Ljava/lang/Object;)V (Actor.java:32)
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks()V (ProcessorBase.java:68)
> at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$548+0x00007f62585c4d80.run()V ()
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1136)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:635)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run()V (ActiveMQThreadFactory.java:118) {code}
>
> Please suggest on how to handle against barrage of incoming connections
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact