Strategy question - many queues or use selectors?

[James Green <ja...@gmail.com>]

I'm sure this must get asked routinely but for my benefit at least,
which is considered best: Have many (hundreds) of queues, or a single
queue of similar messages and use selectors to pick the right ones
out?

To shed a little more light on my scenario we have an application to
which several hundred clients connect via a web interface. This
interface performs the authentication and authorisation then connects
to AMQ. It subscribes currently to a queue called "Outbound.Account.X"
where X is an account number. It actually uses selectors to guarantee
ordering of three message types already. This appears to work
"adequately".

Another way might be to maintain a single queue "Outbound" and mark up
the messages with an account number in addition to the existing
headers. We would then see multiple connections to the same queue each
using various selectors to get the individual account's messages in
the correct order.

Is there any advice or evidence to suggest one strategy is better than
the other? Clearly there will be implementation-specific pros & cons
for AMQ itself but I'm also interested in case people have experience
of one strategy working better than the other (perhaps from an
administrative perspective even).

Thanks,

James

Re: Strategy question - many queues or use selectors?

[Mats Henrikson <ma...@zakalwe.com>]

Hi James,

On 12 April 2011 07:54, James Green <ja...@gmail.com> wrote:
> I'm sure this must get asked routinely but for my benefit at least,
> which is considered best: Have many (hundreds) of queues, or a single
> queue of similar messages and use selectors to pick the right ones
> out?

We have in the order of thousands of queues and topics, and many of
the destinations also have possibly up to 10 selectors. My experience
on ActiveMQ 5.2 and 5.3 was atrocious as there were problems with
controlling the number of threads, so we would easily run out of
threads. Since upgrading to 5.4 all the thread controls appear to be
functioning well (but we could do with having SSL over NIO implemented
to make things even better).

Unfortunately, selectors have been and are still problematic due to
the interaction with maxPageSize, which as far as I know isn't
scheduled to be fixed until ActiveMQ 6.0.

For the reasons above I would definitely go with a
multiple-destinations-no-selectors approach if I could start our
project all over again, as opposed to a
fewer-destinations-multiple-selectors approach.

Good luck,

Mats

Re: Strategy question - many queues or use selectors?

[Dejan Bosanac <de...@nighttale.net>]

Hi James,

one thing to think about is optimization of memory vs. threads.

If you're going with large number of destinations and consumers be sure to
check activemq-scalability.xml example and read more on a broker scalability
(
http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html)
and controling number of threads.

In case of one destination, the problem is that you can have very deep
queues and selectors don't work on the whole queue size, but only on those
messages cached in the memory. So you'll have to have a large page size to
keep all messages in the memory.

I'd go with the first option as your system scale you can also partition
destinations and client among multiple brokers with the little work on your
client side.

Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net

Connect at CamelOne <http://camelone.com/> May 24-26

The Open Source Integration Conference



On Mon, Apr 11, 2011 at 9:54 PM, James Green <ja...@gmail.com>wrote:

> I'm sure this must get asked routinely but for my benefit at least,
> which is considered best: Have many (hundreds) of queues, or a single
> queue of similar messages and use selectors to pick the right ones
> out?
>
> To shed a little more light on my scenario we have an application to
> which several hundred clients connect via a web interface. This
> interface performs the authentication and authorisation then connects
> to AMQ. It subscribes currently to a queue called "Outbound.Account.X"
> where X is an account number. It actually uses selectors to guarantee
> ordering of three message types already. This appears to work
> "adequately".
>
> Another way might be to maintain a single queue "Outbound" and mark up
> the messages with an account number in addition to the existing
> headers. We would then see multiple connections to the same queue each
> using various selectors to get the individual account's messages in
> the correct order.
>
> Is there any advice or evidence to suggest one strategy is better than
> the other? Clearly there will be implementation-specific pros & cons
> for AMQ itself but I'm also interested in case people have experience
> of one strategy working better than the other (perhaps from an
> administrative perspective even).
>
> Thanks,
>
> James
>