RE: Message Selectors vs multiple queues

[John Lilley <jo...@redpointglobal.com.INVALID>]

Awesome, that’s a very good metric to know.
--john




[rg] <https://www.redpointglobal.com/>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Friday, December 30, 2022 12:46 PM
To: users@activemq.apache.org
Subject: Re: Message Selectors vs multiple queues

*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***

> How much memory does each queue require of the broker?

I just ran a quick test with a clean, default instance of 2.27.1 where I forced GC, inspected the heap size, created 1,000 durable anycast queues (which require 1,000 addresses as well), forced GC, then inspected the heap size again. When the test started the broker was using around 24MB of heap. After the test finished it was using around 560MB of heap. That means each address+queue is taking about 550KB of heap (i.e. (560-24)*1024/1000).

> What is the scanning overhead and when does it really become an issue? If we have a mostly-empty queue and handling 100/second requests is that going to be OK? If we have a queue with 100 messages backed up and 1000/second does that start to tip over the edge and become too slow?

I can't really answer these questions as it's based on your specific use-case and performance goals.


Justin

On Thu, Dec 22, 2022 at 7:49 AM John Lilley <jo...@redpointglobal.com.invalid>> wrote:

Mark,

Yeah that's kind of where I'm coming from. We've seen that a large number of queues consumes a lot of broker memory, and we've hit OOM a few times due to 100s to reply-to temporary queues. We've since fixed the reply-to queue count by having our RPC receivers multiplex a single reply-to queue. However, we now face a similar tradeoff when it comes to our "job" RPC queues, which are fairly low traffic (each job might get 5/second messages). It would also be nice to multiplex all job RPC comms through a single queue, because then if a job crashes we can have a "last chance" service take over handling of all those jobs' requests and return errors. We currently do so but we end up with a "last chance" service potentially keeping 100s of queues active for a while "just in case" someone tries to communicate with a crashed job. And that chews up broker memory, forcing us to reduce the "last chance" responder time window.

So scalability and throughput are not the only things driving this decision. We also want to reduce the queue count to limit memory pressure on the broker, and make it easier to cover for crashed jobs.

I would really like know:
-- How much memory does each queue require of the broker?
-- What is the scanning overhead and when does it really become an issue? If we have a mostly-empty queue and handling 100/second requests is that going to be OK? If we have a queue with 100 messages backed up and 1000/second does that start to tip over the edge and become too slow?

Thanks
John


[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,Vd4cUyDDgSHpeD7Qx7w5RMJwNVGEXdXV07ZS_0WjEBMdqaYW48Q0H7Bd_D9vppnL5d6B3WVZQD_ksY_unl389aKHCntJOARMH4wIeK6QAVn0-rQ,&typo=1>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>

-----Original Message-----
From: Mark Johnson <sp...@gmail.com>>
Sent: Thursday, December 22, 2022 5:35 AM
To: users@activemq.apache.org<ma...@activemq.apache.org>
Subject: Re: Message Selectors vs multiple queues

*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***

Hi Justin,

Is there a rule of thumb for the limit of Y?

Just wondering about the realistic upper limit on the number of queues.

Thanks

On Wed, 21 Dec 2022 at 15:21, Justin Bertram <jb...@apache.org>> wrote:

> Generally speaking I would avoid use-cases involving a JMS queue +
> selectors when possible for the following reasons:
>
> - The queue has to be scanned over and over for messages which match
> the selectors. The deeper the queue gets the more scanning is
> required. This adds up over time.
> - A single queue with X traffic will almost certainly not be as fast
> as Y queues with X/Y traffic on each. Put simplistically, more
> concurrency equals more performance both in terms of throughput and scalability.
>
>
> Justin
>
> On Tue, Dec 20, 2022 at 5:26 PM John Lilley
> <jo...@redpointglobal.com.invalid>> wrote:
>
>> Greetings!
>>
>>
>>
>> We have a design wherein every “job” that runs offers up an RPC
>> service via a unique queue name. Which means that if 100 jobs are
>> running, there are 100 queues. I’ve read that, in theory, all jobs
>> could share a single queue and use Message Selectors (in JMS) to
>> receive only messages that they should service. Can anyone help me
>> understand the tradeoffs? Would shared queue tend to use less broker
>> memory? Would the shared queue performance start to degrade under load?
>>
>>
>>
>> Thanks,
>>
>> John
>>
>>
>>
>> [image: rg]
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglob
>> al.com<http://al.com>%2f&c=E,1,GWRpQJge7Z3JyOqH7F7YViXtCc3aUUWoOcyrofUIjoN0S_pkWLJjw
>> lLvrSeL-eorBWmJHJMwzHTmimlWsOH2WKdMQwd3GYf1io4GYNQV4yuXD9C8xKzFtQ,,&t
>> ypo=1>
>>
>> John Lilley
>>
>> Data Management Chief Architect, Redpoint Global Inc.
>>
>> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>>
>> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
>>
>> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
>> confidential and is intended solely for the use of the individual(s)
>> to whom it is addressed. If you believe you received this e-mail in
>> error, please notify the sender immediately, delete the e-mail from
>> your computer and do not copy, print or disclose it to anyone else.
>> If you properly received this e-mail as a customer, partner or vendor
>> of Redpoint, you should maintain its contents in confidence subject
>> to the terms and conditions of your agreement(s) with Redpoint.
>>
>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.