[jira] [Comment Edited] (AMQ-9107) Closing many consumers causes CPU to spike to 100%

["Christopher L. Shannon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627262#comment-17627262 ] 

Christopher L. Shannon edited comment on AMQ-9107 at 11/1/22 5:22 PM:
----------------------------------------------------------------------

I created a draft PR here: https://github.com/apache/activemq/pull/928

I'm still not sure if it will work entirely yet because of the fact that I think multiple subscriptions can be attached to one ConsumerInfo object (different destinations). I covered the composite case but not sure if there's other cases without looking at it more. But the fact that the previous commit was iterating through all subscriptions to find matching ConsumerInfo objects tells me that it's probably not always a 1 to 1 mapping.

I have to step away for a bit but can look more later today.


was (Author: christopher.l.shannon):
I created a draft PR here: https://github.com/apache/activemq/pull/928

I'm still not sure if it will work entirely yet because of the fact that I think multiple subscriptions can be attached to one ConsumerInfo object (different destinations). I covered the composite case but not sure if there's other cases without looking at it more. But the fact that the previous commit was iterating through all subscriptions to find matching ConsumerInfo objects tells me that it's probably not always a 1 to 1 mapping.

> Closing many consumers causes CPU to spike to 100%
> --------------------------------------------------
>
>                 Key: AMQ-9107
>                 URL: https://issues.apache.org/jira/browse/AMQ-9107
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.17.1, 5.16.5
>            Reporter: Lucas Tétreault
>            Assignee: Jean-Baptiste Onofré
>            Priority: Major
>             Fix For: 5.18.0, 5.16.6, 5.17.3
>
>         Attachments: example.zip, image-2022-10-07-00-12-39-657.png, image-2022-10-07-00-17-30-657.png
>
>          Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> When there are many consumers (~188k) on a queue, closing them is incredibly expensive and causes the CPU to spike to 100% while the consumers are closed. Tested on an Amazon MQ mq.m5.large instance (2 vcpu, 8gb memory).
> I have attached a minimal recreation of the issue where the following happens: 
> 1/ Open 100 connections.
> 2/ Create consumers as fast as we can on all of those connections until we hit at least 188k consumers.
> 3/ Sleep for 5 minutes so we can observe the CPU come back down after opening all those connections.
> 4/ Start closing consumers as fast as we can.
> 5/ After all consumers are closed, sleep for 5 minutes to observe the CPU come back down after closing all the connections.
>  
> In this example it seems 5 minutes wasn't actually sufficient time for the CPU to come back down and the consumer and connection counts seem to hit 0 at the same time: 
> !image-2022-10-07-00-12-39-657.png|width=757,height=353!
>  
> In a previous test with more time sleeping after closing all the consumers we can see the CPU come back down before we close the connections. 
> !image-2022-10-07-00-17-30-657.png|width=764,height=348!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)