Multiple consumers serving multiple queues

[chicagoclimber <ch...@gmail.com>]

Hi, 

I have read a lot of documentation over the past few days trying to answer
what I thought was a very standard question: I have a farm of consumers
(50-100) that needs to process messages from multiple sources (10-20).
Messages from some of these sources will be coming in bursts, and I need to
make sure that this won't affect messages from other sources. In other words
I need to implement some sort of flow control logic.

I can imagine two solutions to these problem:

Solution 1. Multiple input queues are all routed into one output queue. All
consumers subscribe to this output queue. This output queue will use
producer flow control to make sure that it remains small.

There are several problems with this solution:

 * Flow control only kicks in when the output queue is full, so this queue
would have to be really small so as not to introduce a lot of latency for
messages from other sources. This would probably introduce a lot of
contention.
  * This flow control is completely symmetric for all producers, so I can't
treat messages from one producer as more important than messages from
another producer.
  * Finally it looks like flow control in ActiveMQ is only used with
non-durable queues. If that's indeed true then I can't use this approach.

I do know about time-based throttling, but it won't work for me since it is
completely agnostic of the actual message load across all queues.

Solution 2. All consumers subscribe to all queues (using wildcard
mechanism). Each queue in ActiveMQ has a "pending queue" (pagedInMessages).
Controlling the maximum size of this pending queue I can assign relative
priorities for different queues. 

This approach seems to have very nice properties, however I am concerned
about performance implications of using small maxPageSize, as well as using
having each consumer subscribe to so many queues.
 
Has anyone done anything similar before?

Any advice will be greatly appreciated.

Thanks!
-sergey
-- 
View this message in context: http://www.nabble.com/Multiple-consumers-serving-multiple-queues-tp16765830s2354p16765830.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.