You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by John Lilley <jo...@redpointglobal.com.INVALID> on 2022/11/04 19:15:55 UTC
Practical limit on #queues in Artemis
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Practical limit on #queues in Artemis
Posted by Justin Bertram <jb...@apache.org>.
> With regard to experienced slowdown, of the management console I assume,
does that get better if you apply name-based filtering to the views?
The problem isn't typically with views where name-based filtering is an
option as those views are already paged to avoid large datasets. The
problem is usually with the "tree" view on the left. To build this view the
browser asks the Jolokia agent running in the broker's JVM to build a JSON
payload including potentially *every* MBean (including all of the
attributes for each MBean). If you have, for example, 3,000 anycast queues
each on their own address the JSON payload is going to contain the data for
6,000 MBeans. By default this data is refreshed every 5 seconds. You can
quickly see how this would become a problem.
That said, the web console's behavior is configurable. Go to the
"Preferences" (available from the menu in the top right) and click the
"Jolokia" tab. Here you can turn off auto-refresh (i.e. "Update rate"). You
can also decrease the amount of data fetched by lowering the "Max depth"
and "Max collection size."
Justin
On Sat, Nov 12, 2022 at 10:31 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Wow, that is great. Thanks!
>
>
>
> With regard to experienced slowdown, of the management console I assume,
> does that get better if you apply name-based filtering to the views?
>
>
>
> Thanks
>
> John
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Roskvist Anton <an...@volvo.com>
> *Sent:* Thursday, November 10, 2022 12:32 PM
> *To:* users@activemq.apache.org
> *Subject:* RE: Practical limit on #queues in Artemis
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Hello John,
>
>
>
> For reference, one of the environments I’m maintaining currently uses
> around 3.5k active destinations. The broker cluster is running on
> relatively modest servers without any issues. I will say though that at
> this point there has been some noticeable slowdowns in precisely the areas
> that Justin mentioned… that being said the clients (running on separate
> servers) report an average publish latency of ~0.8ms on the actual queues.
> That’s quite impressive I think.
>
>
>
> Br,
>
> Anton
>
>
>
> *From:* John Lilley <jo...@redpointglobal.com.INVALID>
> *Sent:* den 9 november 2022 15:56
> *To:* users@activemq.apache.org
> *Subject:* RE: Practical limit on #queues in Artemis
>
>
>
> CAUTION: This email originated from outside of the organization. If
> suspicious, please report it.
>
> Thanks Justin!
>
> I’ll run some tests and just see what happens.
>
> John
>
>
>
>
>
>
>
> [image: rg]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyuoSBmj%2FvQSF0NPHcV2Tb8qKqf04Y%2Bpr44iSA8aTVs%3D&reserved=0>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Tuesday, November 8, 2022 10:04 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: Practical limit on #queues in Artemis
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> There is certainly a practical limit, but I doubt anybody has a clear view
> of what it is. I think you're more likely to hit pain points with
> management (e.g. the web console) before the queues themselves cause
> problems.
>
>
>
>
>
> Justin
>
>
>
> On Fri, Nov 4, 2022 at 2:16 PM John Lilley <
> john.lilley@redpointglobal.com.invalid> wrote:
>
> Greetings,
>
>
> We create a queue per “job” and there can be a lot of jobs. We tend to
> keep them around for a while (like an hour) to service job status after the
> job is complete, but this can result in a lot of mostly-idle queues. Is
> there a practical limit at which Artemis becomes unhappy (low memory,
> sluggish performance)? I’m sure it depends on VM sizing, but let’s just
> say that we have Artemis running in a K8S pod with 4GB memory available and
> 4vcpus.
>
>
>
> I’m only looking for an order-of-magnitude ballpark. Something like “no
> more than N per GB memory” would be great.
>
>
>
> Thanks
>
> John
>
>
>
>
>
> [image: rg]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4wyMy4eNE3hI2zABvg1oHV7MXxwkzkfgo9nyfNAbj4I%3D&reserved=0>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
>
> *This email message (including its attachments) is confidential and may
> contain privileged information and is intended solely for the use of the
> individual and/or entity to whom it is addressed. If you are not the
> intended recipient of this e-mail you may not disseminate, distribute or
> copy this e-mail (including its attachments), or any part thereof. If this
> e-mail is received in error, please notify the sender immediately by return
> e-mail and make sure that this e-mail (including its attachments), and all
> copies thereof, are immediately deleted from your system. Please further
> note that when you communicate with us via email or visit our website we
> process your personal data. See our privacy policy for more information
> about how we process it: https://www.volvogroup.com/en-en/privacy.html
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.volvogroup.com%2fen-en%2fprivacy.html&c=E,1,1l_pi5EBgs-jikH5_gANZQ-bFwp2PBOJDaN57Z7QEA8Gbt6Rq_xt2w-bFGe3Gc6vNy593p_ghkaNRtE5j1OiuUoFb4oSn1dj-2RyFDmkhrYzGHHN0Yj0&typo=1&ancr_add=1>
> *
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
RE: Practical limit on #queues in Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Thanks!
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Roskvist Anton <an...@volvo.com>
Sent: Monday, November 14, 2022 1:48 AM
To: users@activemq.apache.org
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
That particular cluster is running on AWS m6a.2xlarge ec2-instances. It’s a static cluster consisting of 3 nodes in total. There might be instance types better suited for their particular use case such as r6a.xlarge or something similar though, not sure.
As for the slowdown, yes, that’s for the web console and collecting metrics through jolokia. We actually switched from using Jolokia to using a metric plugin based on: https://github.com/rh-messaging/artemis-prometheus-metrics-plugin
The metric plugin is very responsive so no issues there. To be honest we do not use the web console all that much and so I cannot tell you if name-based filtering speed things up or not as we have not really looked in to it. It’s far from unusable or anything like that so for our purposes it does not matter that loading up pages takes a few seconds. Let me know if you find anything good there though 😊
If there’s one thing I would recommend for these volumes it would be the ZGC garbage collector. It really keeps things responsive with its sub 1ms collection times.
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: den 12 november 2022 18:00
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
One more thing… how big are your “relatively modest” servers? Because one person’s modest is another’s profligate 😊
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlGoxRXVKGO55xTfRrcGV8eay4oxcEjN%2F65YBuOX1fc%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: Saturday, November 12, 2022 9:31 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Wow, that is great. Thanks!
With regard to experienced slowdown, of the management console I assume, does that get better if you apply name-based filtering to the views?
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2CvGfGvHh2t4xJN-hB5uQsNa3NtJKmpiAApZiiclQohe-KAcQkaPqWYgsgMA7-pK2fHy5D74zOdJm1A-o_ncTnaiAR6sJHKSJMsukClQDRFIemXEk%2C%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OPqctpVx7s1OFfRqmgbtdsR0m0sZLR7BoeZnN25%2Bmuo%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Roskvist Anton <an...@volvo.com>>
Sent: Thursday, November 10, 2022 12:32 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hello John,
For reference, one of the environments I’m maintaining currently uses around 3.5k active destinations. The broker cluster is running on relatively modest servers without any issues. I will say though that at this point there has been some noticeable slowdowns in precisely the areas that Justin mentioned… that being said the clients (running on separate servers) report an average publish latency of ~0.8ms on the actual queues. That’s quite impressive I think.
Br,
Anton
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: den 9 november 2022 15:56
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
CAUTION: This email originated from outside of the organization. If suspicious, please report it.
Thanks Justin!
I’ll run some tests and just see what happens.
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlGoxRXVKGO55xTfRrcGV8eay4oxcEjN%2F65YBuOX1fc%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8s9%2FgVrPmG8O65d1AjmZTQAaEB7rcsUt79jp3VQyVOA%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.volvogroup.com%252fen-en%252fprivacy.html%26c%3DE%2C1%2C1l_pi5EBgs-jikH5_gANZQ-bFwp2PBOJDaN57Z7QEA8Gbt6Rq_xt2w-bFGe3Gc6vNy593p_ghkaNRtE5j1OiuUoFb4oSn1dj-2RyFDmkhrYzGHHN0Yj0%26typo%3D1%26ancr_add%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JMs69kdtkEpg6WlzvgEbmnRf%2FejTgAgGASyJT1MAN%2FQ%3D&reserved=0>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.volvogroup.com%2fen-en%2fprivacy.html&c=E,1,iLHa5k1s61x2cAOSwRZW_ZDkE7nKQ6oWDYvmb2bMgYYxO8JuowzJXSfDirGiLfnmsglmBmI608PCKciVY07G3hNjKTfhCbplWDnf6DzXF6g-_D4usFc,&typo=1&ancr_add=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Practical limit on #queues in Artemis
Posted by Roskvist Anton <an...@volvo.com>.
That particular cluster is running on AWS m6a.2xlarge ec2-instances. It’s a static cluster consisting of 3 nodes in total. There might be instance types better suited for their particular use case such as r6a.xlarge or something similar though, not sure.
As for the slowdown, yes, that’s for the web console and collecting metrics through jolokia. We actually switched from using Jolokia to using a metric plugin based on: https://github.com/rh-messaging/artemis-prometheus-metrics-plugin
The metric plugin is very responsive so no issues there. To be honest we do not use the web console all that much and so I cannot tell you if name-based filtering speed things up or not as we have not really looked in to it. It’s far from unusable or anything like that so for our purposes it does not matter that loading up pages takes a few seconds. Let me know if you find anything good there though 😊
If there’s one thing I would recommend for these volumes it would be the ZGC garbage collector. It really keeps things responsive with its sub 1ms collection times.
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: den 12 november 2022 18:00
To: users@activemq.apache.org
Subject: RE: Practical limit on #queues in Artemis
One more thing… how big are your “relatively modest” servers? Because one person’s modest is another’s profligate 😊
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlGoxRXVKGO55xTfRrcGV8eay4oxcEjN%2F65YBuOX1fc%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: Saturday, November 12, 2022 9:31 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Wow, that is great. Thanks!
With regard to experienced slowdown, of the management console I assume, does that get better if you apply name-based filtering to the views?
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2CvGfGvHh2t4xJN-hB5uQsNa3NtJKmpiAApZiiclQohe-KAcQkaPqWYgsgMA7-pK2fHy5D74zOdJm1A-o_ncTnaiAR6sJHKSJMsukClQDRFIemXEk%2C%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OPqctpVx7s1OFfRqmgbtdsR0m0sZLR7BoeZnN25%2Bmuo%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Roskvist Anton <an...@volvo.com>>
Sent: Thursday, November 10, 2022 12:32 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hello John,
For reference, one of the environments I’m maintaining currently uses around 3.5k active destinations. The broker cluster is running on relatively modest servers without any issues. I will say though that at this point there has been some noticeable slowdowns in precisely the areas that Justin mentioned… that being said the clients (running on separate servers) report an average publish latency of ~0.8ms on the actual queues. That’s quite impressive I think.
Br,
Anton
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: den 9 november 2022 15:56
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
CAUTION: This email originated from outside of the organization. If suspicious, please report it.
Thanks Justin!
I’ll run some tests and just see what happens.
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlGoxRXVKGO55xTfRrcGV8eay4oxcEjN%2F65YBuOX1fc%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8s9%2FgVrPmG8O65d1AjmZTQAaEB7rcsUt79jp3VQyVOA%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.volvogroup.com%252fen-en%252fprivacy.html%26c%3DE%2C1%2C1l_pi5EBgs-jikH5_gANZQ-bFwp2PBOJDaN57Z7QEA8Gbt6Rq_xt2w-bFGe3Gc6vNy593p_ghkaNRtE5j1OiuUoFb4oSn1dj-2RyFDmkhrYzGHHN0Yj0%26typo%3D1%26ancr_add%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7C6cb7c03828f642b2350608dac4cf6f6e%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638038692453443966%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JMs69kdtkEpg6WlzvgEbmnRf%2FejTgAgGASyJT1MAN%2FQ%3D&reserved=0>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html
RE: Practical limit on #queues in Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
One more thing… how big are your “relatively modest” servers? Because one person’s modest is another’s profligate 😊
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: Saturday, November 12, 2022 9:31 AM
To: users@activemq.apache.org
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Wow, that is great. Thanks!
With regard to experienced slowdown, of the management console I assume, does that get better if you apply name-based filtering to the views?
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,vGfGvHh2t4xJN-hB5uQsNa3NtJKmpiAApZiiclQohe-KAcQkaPqWYgsgMA7-pK2fHy5D74zOdJm1A-o_ncTnaiAR6sJHKSJMsukClQDRFIemXEk,&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Roskvist Anton <an...@volvo.com>>
Sent: Thursday, November 10, 2022 12:32 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hello John,
For reference, one of the environments I’m maintaining currently uses around 3.5k active destinations. The broker cluster is running on relatively modest servers without any issues. I will say though that at this point there has been some noticeable slowdowns in precisely the areas that Justin mentioned… that being said the clients (running on separate servers) report an average publish latency of ~0.8ms on the actual queues. That’s quite impressive I think.
Br,
Anton
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: den 9 november 2022 15:56
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
CAUTION: This email originated from outside of the organization. If suspicious, please report it.
Thanks Justin!
I’ll run some tests and just see what happens.
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyuoSBmj%2FvQSF0NPHcV2Tb8qKqf04Y%2Bpr44iSA8aTVs%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4wyMy4eNE3hI2zABvg1oHV7MXxwkzkfgo9nyfNAbj4I%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.volvogroup.com%2fen-en%2fprivacy.html&c=E,1,1l_pi5EBgs-jikH5_gANZQ-bFwp2PBOJDaN57Z7QEA8Gbt6Rq_xt2w-bFGe3Gc6vNy593p_ghkaNRtE5j1OiuUoFb4oSn1dj-2RyFDmkhrYzGHHN0Yj0&typo=1&ancr_add=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Practical limit on #queues in Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Wow, that is great. Thanks!
With regard to experienced slowdown, of the management console I assume, does that get better if you apply name-based filtering to the views?
Thanks
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Roskvist Anton <an...@volvo.com>
Sent: Thursday, November 10, 2022 12:32 PM
To: users@activemq.apache.org
Subject: RE: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
Hello John,
For reference, one of the environments I’m maintaining currently uses around 3.5k active destinations. The broker cluster is running on relatively modest servers without any issues. I will say though that at this point there has been some noticeable slowdowns in precisely the areas that Justin mentioned… that being said the clients (running on separate servers) report an average publish latency of ~0.8ms on the actual queues. That’s quite impressive I think.
Br,
Anton
From: John Lilley <jo...@redpointglobal.com.INVALID>>
Sent: den 9 november 2022 15:56
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: RE: Practical limit on #queues in Artemis
CAUTION: This email originated from outside of the organization. If suspicious, please report it.
Thanks Justin!
I’ll run some tests and just see what happens.
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyuoSBmj%2FvQSF0NPHcV2Tb8qKqf04Y%2Bpr44iSA8aTVs%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4wyMy4eNE3hI2zABvg1oHV7MXxwkzkfgo9nyfNAbj4I%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.volvogroup.com%2fen-en%2fprivacy.html&c=E,1,1l_pi5EBgs-jikH5_gANZQ-bFwp2PBOJDaN57Z7QEA8Gbt6Rq_xt2w-bFGe3Gc6vNy593p_ghkaNRtE5j1OiuUoFb4oSn1dj-2RyFDmkhrYzGHHN0Yj0&typo=1&ancr_add=1>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
RE: Practical limit on #queues in Artemis
Posted by Roskvist Anton <an...@volvo.com>.
Hello John,
For reference, one of the environments I'm maintaining currently uses around 3.5k active destinations. The broker cluster is running on relatively modest servers without any issues. I will say though that at this point there has been some noticeable slowdowns in precisely the areas that Justin mentioned... that being said the clients (running on separate servers) report an average publish latency of ~0.8ms on the actual queues. That's quite impressive I think.
Br,
Anton
From: John Lilley <jo...@redpointglobal.com.INVALID>
Sent: den 9 november 2022 15:56
To: users@activemq.apache.org
Subject: RE: Practical limit on #queues in Artemis
CAUTION: This email originated from outside of the organization. If suspicious, please report it.
Thanks Justin!
I'll run some tests and just see what happens.
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redpointglobal.com%2F&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyuoSBmj%2FvQSF0NPHcV2Tb8qKqf04Y%2Bpr44iSA8aTVs%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per "job" and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I'm sure it depends on VM sizing, but let's just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I'm only looking for an order-of-magnitude ballpark. Something like "no more than N per GB memory" would be great.
Thanks
John
[rg]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkprotect.cudasvc.com%2Furl%3Fa%3Dhttps%253a%252f%252fwww.redpointglobal.com%252f%26c%3DE%2C1%2Ca8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql%26typo%3D1&data=05%7C01%7Canton.roskvist%40volvo.com%7Ca30d60c4defd4257fa7708dac262a8ec%7Cf25493ae1c9841d78a330be75f5fe603%7C0%7C0%7C638036026240483171%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4wyMy4eNE3hI2zABvg1oHV7MXxwkzkfgo9nyfNAbj4I%3D&reserved=0>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
This email message (including its attachments) is confidential and may contain privileged information and is intended solely for the use of the individual and/or entity to whom it is addressed. If you are not the intended recipient of this e-mail you may not disseminate, distribute or copy this e-mail (including its attachments), or any part thereof. If this e-mail is received in error, please notify the sender immediately by return e-mail and make sure that this e-mail (including its attachments), and all copies thereof, are immediately deleted from your system. Please further note that when you communicate with us via email or visit our website we process your personal data. See our privacy policy for more information about how we process it: https://www.volvogroup.com/en-en/privacy.html
RE: Practical limit on #queues in Artemis
Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Thanks Justin!
I’ll run some tests and just see what happens.
John
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Tuesday, November 8, 2022 10:04 PM
To: users@activemq.apache.org
Subject: Re: Practical limit on #queues in Artemis
*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***
There is certainly a practical limit, but I doubt anybody has a clear view of what it is. I think you're more likely to hit pain points with management (e.g. the web console) before the queues themselves cause problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,
We create a queue per “job” and there can be a lot of jobs. We tend to keep them around for a while (like an hour) to service job status after the job is complete, but this can result in a lot of mostly-idle queues. Is there a practical limit at which Artemis becomes unhappy (low memory, sluggish performance)? I’m sure it depends on VM sizing, but let’s just say that we have Artemis running in a K8S pod with 4GB memory available and 4vcpus.
I’m only looking for an order-of-magnitude ballpark. Something like “no more than N per GB memory” would be great.
Thanks
John
[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,a8aL6_GO7_wvHtc4lTsWrsJVcK2R3i5wfLc7ihagaX-6tQm79faPPdZ48z0APtj7ZSpJvkDiSterVPAT44gVh4dMlx31S0lajTRltfql&typo=1>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
Re: Practical limit on #queues in Artemis
Posted by Justin Bertram <jb...@apache.org>.
There is certainly a practical limit, but I doubt anybody has a clear view
of what it is. I think you're more likely to hit pain points with
management (e.g. the web console) before the queues themselves cause
problems.
Justin
On Fri, Nov 4, 2022 at 2:16 PM John Lilley
<jo...@redpointglobal.com.invalid> wrote:
> Greetings,
>
>
> We create a queue per “job” and there can be a lot of jobs. We tend to
> keep them around for a while (like an hour) to service job status after the
> job is complete, but this can result in a lot of mostly-idle queues. Is
> there a practical limit at which Artemis becomes unhappy (low memory,
> sluggish performance)? I’m sure it depends on VM sizing, but let’s just
> say that we have Artemis running in a K8S pod with 4GB memory available and
> 4vcpus.
>
>
>
> I’m only looking for an order-of-magnitude ballpark. Something like “no
> more than N per GB memory” would be great.
>
>
>
> Thanks
>
> John
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>