load balance topic consumer

[igah <we...@mail.com>]

hi there,

it's easy to load balance a queue's consumers -- by starting multiple
consumer threads (in the same or different processes). but how to do this
for a subscriber. 

let me explain what I mean with a use case like this: a message is published
to a topic, where there are two subscribers, each of which does something
different. let's say one subscriber's message processing rate is slower than
the message arrival rate. it's natural to start multiple of them to
distribute the load. but you cannot start another identical instance of the
subscriber because that would cause the same message to be processed twice. 

i can think of one solution where i create another queue and forward all the
messages from the topic to the queue. then we can start as many consumers on
that queue as we want to. this requires management of this "forwarder". (you
need to make sure it's always up, etc.) 

is there a better or more "out-of-box" approach? 

thanks in advance.  
-- 
View this message in context: http://www.nabble.com/load-balance-topic-consumer-tf3139637.html#a8702320
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: load balance topic consumer

[James Strachan <ja...@gmail.com>]

On 1/30/07, igah <we...@mail.com> wrote:
>
> that's precisely what i was looking for. thanks!
>
> now the naming convention seems a little bit odd. if i have
> arbitrary/fixed/predefined names for topic and queues (say foo and bar), can
> i link them up?

You can explicitly configure in the activemq.xml a virtual topic to
map to a number of physical queues if you want - otherwise the default
out of the box naming convention will avoid you having ot mess with
XML

James

>
>
> rajdavies wrote:
> >
> > are you looking for this: http://www.activemq.org/site/virtual-
> > destinations.html
> >
> > On 30 Jan 2007, at 02:31, igah wrote:
> >
> >>
> >> hi there,
> >>
> >> it's easy to load balance a queue's consumers -- by starting multiple
> >> consumer threads (in the same or different processes). but how to
> >> do this
> >> for a subscriber.
> >>
> >> let me explain what I mean with a use case like this: a message is
> >> published
> >> to a topic, where there are two subscribers, each of which does
> >> something
> >> different. let's say one subscriber's message processing rate is
> >> slower than
> >> the message arrival rate. it's natural to start multiple of them to
> >> distribute the load. but you cannot start another identical
> >> instance of the
> >> subscriber because that would cause the same message to be
> >> processed twice.
> >>
> >> i can think of one solution where i create another queue and
> >> forward all the
> >> messages from the topic to the queue. then we can start as many
> >> consumers on
> >> that queue as we want to. this requires management of this
> >> "forwarder". (you
> >> need to make sure it's always up, etc.)
> >>
> >> is there a better or more "out-of-box" approach?
> >>
> >> thanks in advance.
> >> --
> >> View this message in context: http://www.nabble.com/load-balance-
> >> topic-consumer-tf3139637.html#a8702320
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
>
> --
> View this message in context: http://www.nabble.com/load-balance-topic-consumer-tf3139637.html#a8703850
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 

James
-------
http://radio.weblogs.com/0112098/

Re: load balance topic consumer

[igah <we...@mail.com>]

that's precisely what i was looking for. thanks!

now the naming convention seems a little bit odd. if i have
arbitrary/fixed/predefined names for topic and queues (say foo and bar), can
i link them up? 


rajdavies wrote:
> 
> are you looking for this: http://www.activemq.org/site/virtual-
> destinations.html
> 
> On 30 Jan 2007, at 02:31, igah wrote:
> 
>>
>> hi there,
>>
>> it's easy to load balance a queue's consumers -- by starting multiple
>> consumer threads (in the same or different processes). but how to  
>> do this
>> for a subscriber.
>>
>> let me explain what I mean with a use case like this: a message is  
>> published
>> to a topic, where there are two subscribers, each of which does  
>> something
>> different. let's say one subscriber's message processing rate is  
>> slower than
>> the message arrival rate. it's natural to start multiple of them to
>> distribute the load. but you cannot start another identical  
>> instance of the
>> subscriber because that would cause the same message to be  
>> processed twice.
>>
>> i can think of one solution where i create another queue and  
>> forward all the
>> messages from the topic to the queue. then we can start as many  
>> consumers on
>> that queue as we want to. this requires management of this  
>> "forwarder". (you
>> need to make sure it's always up, etc.)
>>
>> is there a better or more "out-of-box" approach?
>>
>> thanks in advance.
>> -- 
>> View this message in context: http://www.nabble.com/load-balance- 
>> topic-consumer-tf3139637.html#a8702320
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/load-balance-topic-consumer-tf3139637.html#a8703850
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: load balance topic consumer

[Rob Davies <ra...@gmail.com>]

are you looking for this: http://www.activemq.org/site/virtual- 
destinations.html

On 30 Jan 2007, at 02:31, igah wrote:

>
> hi there,
>
> it's easy to load balance a queue's consumers -- by starting multiple
> consumer threads (in the same or different processes). but how to  
> do this
> for a subscriber.
>
> let me explain what I mean with a use case like this: a message is  
> published
> to a topic, where there are two subscribers, each of which does  
> something
> different. let's say one subscriber's message processing rate is  
> slower than
> the message arrival rate. it's natural to start multiple of them to
> distribute the load. but you cannot start another identical  
> instance of the
> subscriber because that would cause the same message to be  
> processed twice.
>
> i can think of one solution where i create another queue and  
> forward all the
> messages from the topic to the queue. then we can start as many  
> consumers on
> that queue as we want to. this requires management of this  
> "forwarder". (you
> need to make sure it's always up, etc.)
>
> is there a better or more "out-of-box" approach?
>
> thanks in advance.
> -- 
> View this message in context: http://www.nabble.com/load-balance- 
> topic-consumer-tf3139637.html#a8702320
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: load balance topic consumer

[Daryl Richter <ng...@comcast.net>]

On 1/29/07 9:31 PM, "igah" <we...@mail.com> wrote:

> 
> hi there,
> 
> it's easy to load balance a queue's consumers -- by starting multiple
> consumer threads (in the same or different processes). but how to do this
> for a subscriber.
> 
> let me explain what I mean with a use case like this: a message is published
> to a topic, where there are two subscribers, each of which does something
> different. let's say one subscriber's message processing rate is slower than
> the message arrival rate. it's natural to start multiple of them to
> distribute the load. but you cannot start another identical instance of the
> subscriber because that would cause the same message to be processed twice.

I believe that JMS flow control can prevent this from occurring.  The
publisher will not send the next message until all subscribers have consumed
it.

> 
> i can think of one solution where i create another queue and forward all the
> messages from the topic to the queue. then we can start as many consumers on
> that queue as we want to. this requires management of this "forwarder". (you
> need to make sure it's always up, etc.)
> 
> is there a better or more "out-of-box" approach?
> 
> thanks in advance.

--
Daryl
http://itsallsemantics.com

"I¹m afraid of the easy stuffŠ its always harder than it seemsŠ"
-- Bill Hampton, 2006