You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Lucas Tétreault (Jira)" <ji...@apache.org> on 2022/10/07 07:20:00 UTC
[jira] [Updated] (AMQ-9107) Closing many consumers causes CPU to spike to 100%
[ https://issues.apache.org/jira/browse/AMQ-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lucas Tétreault updated AMQ-9107:
---------------------------------
Description:
When there are many consumers (~188k) on a queue, closing them is incredibly expensive and causes the CPU to spike to 100% while the consumers are closed. Tested on an Amazon MQ mq.m5.large instance (2 vcpu, 8gb memory).
I have attached a minimal recreation of the issue where the following happens:
1/ Open 100 connections.
2/ Create consumers as fast as we can on all of those connections until we hit at least 188k consumers.
3/ Sleep for 5 minutes so we can observe the CPU come back down after opening all those connections.
4/ Start closing consumers as fast as we can.
5/ After all consumers are closed, sleep for 5 minutes to observe the CPU come back down after closing all the connections.
In this example it seems 5 minutes wasn't actually sufficient time for the CPU to come back down and the consumer count and cpu utilization seem to hit 0 at the same time:
!image-2022-10-07-00-12-39-657.png|width=757,height=353!
In a previous test with more time sleeping after closing all the consumers we can see the CPU come back down before we close the connections.
!image-2022-10-07-00-17-30-657.png|width=764,height=348!
was:
When there are many consumers (~188k) on a queue, closing them is incredibly expensive and causes the CPU to spike to 100% while the consumers are closed. Tested on an Amazon MQ mq.m5.large instance (2 vcpu, 8gb memory).
I have attached a minimal recreation of the issue where the following happens:
1/ Open 100 connections.
2/ Create consumers as fast as we can on all of those connections until we hit at least 188k consumers.
3/ Sleep for 5 minutes so we can observe the CPU come back down after opening all those connections.
4/ Start closing consumers as fast as we can.
5/ After all consumers are closed, sleep for 5 minutes to observe the CPU come back down after closing all the connections.
In this example it seems 5 minutes wasn't actually sufficient time for the CPU to come back down and the consumer count and cpu utilization seem to hit 0 at the same time:
!image-2022-10-07-00-12-39-657.png|width=757,height=353!
In a previous test with more time sleeping after closing all the consumers we can see the CPU come back down before we close the connections.
!image-2022-10-07-00-17-30-657.png|width=983,height=448!
> Closing many consumers causes CPU to spike to 100%
> --------------------------------------------------
>
> Key: AMQ-9107
> URL: https://issues.apache.org/jira/browse/AMQ-9107
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.17.1, 5.16.5
> Reporter: Lucas Tétreault
> Priority: Major
> Attachments: example.zip, image-2022-10-07-00-12-39-657.png, image-2022-10-07-00-17-30-657.png
>
>
> When there are many consumers (~188k) on a queue, closing them is incredibly expensive and causes the CPU to spike to 100% while the consumers are closed. Tested on an Amazon MQ mq.m5.large instance (2 vcpu, 8gb memory).
> I have attached a minimal recreation of the issue where the following happens:
> 1/ Open 100 connections.
> 2/ Create consumers as fast as we can on all of those connections until we hit at least 188k consumers.
> 3/ Sleep for 5 minutes so we can observe the CPU come back down after opening all those connections.
> 4/ Start closing consumers as fast as we can.
> 5/ After all consumers are closed, sleep for 5 minutes to observe the CPU come back down after closing all the connections.
>
> In this example it seems 5 minutes wasn't actually sufficient time for the CPU to come back down and the consumer count and cpu utilization seem to hit 0 at the same time:
> !image-2022-10-07-00-12-39-657.png|width=757,height=353!
>
> In a previous test with more time sleeping after closing all the consumers we can see the CPU come back down before we close the connections.
> !image-2022-10-07-00-17-30-657.png|width=764,height=348!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)