You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by taxingmonk <mk...@webhanger.com> on 2006/11/21 14:16:36 UTC
Message Group Overlaps and Load Balancing
Hi Guys,
I've got a use case where I want to load balance a given queue but
paritioned accross selectors. I am convinced that somewhere between
composite destinations and message groups I can achieve what I'm after.
The use case is that I want to be able to send one message which is load
balanced to a series of receievers which I group together somehow but at the
same time I have multiple groups also receiving the same message.
So In my example I have message M1 and I have Consumers (C1,C2,C3) Group 1
and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
receieved by one member of Group 1, Group 2 and Group 3.
I thought that might be achieveable by using composite destinations but I
didn't want to manage a seperate queue for each group so that I could drop a
message on the bus and have to arrive at all of them without specifying send
(a.group1, a.group2, a.group3), etc, etc.
Furthermore, what would be even better is if this could derived from the
message itself. If we consider a numberline where I can specify a range
from say 1 to 10 and group 1 is interested in 1-3 and group 2 is interested
2-6 and group 3 is interested in 5-10. From there I would like to be able to
say this message is for range 1
It's almost like I want layer selector behaviour accross the top of the
message groups while load balancing :) Ultimatly this allows me to spawn
more consumers if I start to realise that the processes that are handling a
particular range have a large data density, the key thing is that multiple
groups can return responses to the one message, ranging specifying
partitions of data, which may or may not overlap. This I fear may be a
bridge too far.
It should be noted that none of these messages is transactionally important,
nor do I need the total guarentee that two or more consumers in the same
group receieve the same message, although that shouldn't be the common case.
Any ideas?
Kind Regards,
Max
--
View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7471026
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message Group Overlaps and Load Balancing
Posted by James Strachan <ja...@gmail.com>.
So if you want overlap - where 1 message may go to several groups, but
each group processes it once - then multiple queues is the best
solution I think. So either use a consumer to do the routing, or a
broker interceptor.
BTW yeah am still based in Islington :)
On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
>
> Hi James,
>
> To quote: "If you just want load balancing you could use 1 queue with a
> selector
> for each group."
>
> Yes, that'd be fine, the problem is that I want an overlap. Having done
> further testing I realise this is a flawed notion in and of itself, I'm
> inadvertently expecting a queue to duplicate messages for the overlapping
> selectors.
>
> Another way to describe the intending behaviour would be to have a set of
> consumers on a topic where groups of consumers could be made where one
> consumer in the group and only one receives a copy of the message for that
> group. Thus I can build a replicated distributed queue. The net result is
> actually identical to using multiple queues and composite destinations.
>
> I think your suggestion: "So send a message to an input queue then have a
> farm of routers consuming them and routing them to the correct queue for the
> right group to process it." is the closest, when I was first thinking up a
> solution for this using composite destinations I was hoping to hide the
> building of the composite destination from the sender. This seems like a
> nice way to do it.
>
> Thanks James,
> Max
> p.s. Are you in "Norf Lundun" ATM?
>
>
>
>
>
>
> James.Strachan wrote:
> >
> > On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I've got a use case where I want to load balance a given queue but
> >> paritioned accross selectors. I am convinced that somewhere between
> >> composite destinations and message groups I can achieve what I'm after.
> >>
> >> The use case is that I want to be able to send one message which is load
> >> balanced to a series of receievers which I group together somehow but at
> >> the
> >> same time I have multiple groups also receiving the same message.
> >>
> >> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group
> >> 1
> >> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
> >> receieved by one member of Group 1, Group 2 and Group 3.
> >
> > You could just use 3 queues for each group - then you can browse what
> > messages are available for each group & monitor the queue depth etc.
> > Thats the main benefit over using selectors.
> >
> > Note that virtual destinations & composite destinations result in the
> > message going to all of the destinations - in this use case you want
> > it to only go to one of them. So selectors or a custom router is the
> > best solution.
> >
> >
> >> I thought that might be achieveable by using composite destinations but I
> >> didn't want to manage a seperate queue for each group so that I could
> >> drop a
> >> message on the bus and have to arrive at all of them without specifying
> >> send
> >> (a.group1, a.group2, a.group3), etc, etc.
> >>
> >> Furthermore, what would be even better is if this could derived from the
> >> message itself. If we consider a numberline where I can specify a range
> >> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is
> >> interested
> >> 2-6 and group 3 is interested in 5-10. From there I would like to be able
> >> to
> >> say this message is for range 1
> >>
> >> It's almost like I want layer selector behaviour accross the top of the
> >> message groups while load balancing :)
> >
> > Just to be clear - Message Groups are basically all about sticky load
> > balancing. All queues - whether using selectors or not - implement
> > load balancing.
> >
> > If you just want load balancing you could use 1 queue with a selector
> > for each group. Using message groups on top of this should give you
> > sticky load balancing if you need it.
> >
> >
> >> Ultimatly this allows me to spawn
> >> more consumers if I start to realise that the processes that are handling
> >> a
> >> particular range have a large data density, the key thing is that
> >> multiple
> >> groups can return responses to the one message, ranging specifying
> >> partitions of data, which may or may not overlap. This I fear may be a
> >> bridge too far.
> >>
> >> It should be noted that none of these messages is transactionally
> >> important,
> >> nor do I need the total guarentee that two or more consumers in the same
> >> group receieve the same message, although that shouldn't be the common
> >> case.
> >
> > You could write a custom Broker interceptor to implement whatever
> > routing policy you wish - using the message headers / body to route to
> > a different queue based on some rules...
> > http://incubator.apache.org/activemq/interceptors.html
> >
> > Though its a bit more MOM-like to do these kinds of routing policies
> > using a consumer which does the routing for you - which can then be as
> > simple or complex as you wish. So send a message to an input queue
> > then have a farm of routers consuming them and routing them to the
> > correct queue for the right group to process it.
> >
> > Am not sure if you need sticky load balancing from your mail - so
> > maybe just 1 queue with selectors would do the trick?
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7474172
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Message Group Overlaps and Load Balancing
Posted by James Strachan <ja...@gmail.com>.
On 11/27/06, taxingmonk <mk...@webhanger.com> wrote:
>
> Really? How does one do that?
The Virtual Destinations are just simple POJOs you can construct
programatically if you like.
http://incubator.apache.org/activemq/maven/activemq-core/apidocs/org/apache/activemq/broker/region/virtual/package-summary.html
so just create your own VirtualDestinationInterceptor class which can
mutate its virtual destinations whenever it wishes - or customize a
specific VirtualDestination or CompositeQueue etc.
Unless you were talking about the maven2 build :)
http://incubator.apache.org/activemq/building.html
--
James
-------
http://radio.weblogs.com/0112098/
Re: Message Group Overlaps and Load Balancing
Posted by taxingmonk <mk...@webhanger.com>.
Really? How does one do that?
James.Strachan wrote:
>
> On 11/27/06, taxingmonk <mk...@webhanger.com> wrote:
>> Hi James,
>>
>> Yup, this should also work, the only thing that is nice about the
>> previous
>> approach is that programatically I can change the partitioning. Either
>> by
>> providing a wrapper to producing the message (and dynamically defining
>> which
>> queues to send to), or doing the same by consuming the message from the
>> "master" queue and republishing.
>
> Agreed. There's nothing to stop you programatically changing the
> Virtual Destination definitions in the broker at runtime either.
>
>
>> p.s. Built with MVN2 these days?
>
> Yes.
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7564939
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message Group Overlaps and Load Balancing
Posted by James Strachan <ja...@gmail.com>.
On 11/27/06, taxingmonk <mk...@webhanger.com> wrote:
> Hi James,
>
> Yup, this should also work, the only thing that is nice about the previous
> approach is that programatically I can change the partitioning. Either by
> providing a wrapper to producing the message (and dynamically defining which
> queues to send to), or doing the same by consuming the message from the
> "master" queue and republishing.
Agreed. There's nothing to stop you programatically changing the
Virtual Destination definitions in the broker at runtime either.
> p.s. Built with MVN2 these days?
Yes.
--
James
-------
http://radio.weblogs.com/0112098/
Re: Message Group Overlaps and Load Balancing
Posted by taxingmonk <mk...@webhanger.com>.
Hi James,
Yup, this should also work, the only thing that is nice about the previous
approach is that programatically I can change the partitioning. Either by
providing a wrapper to producing the message (and dynamically defining which
queues to send to), or doing the same by consuming the message from the
"master" queue and republishing.
Regards,
Max
p.s. Built with MVN2 these days?
James.Strachan wrote:
>
> On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
>>
>> Hi James,
>>
>> To quote: "If you just want load balancing you could use 1 queue with a
>> selector
>> for each group."
>>
>> Yes, that'd be fine, the problem is that I want an overlap. Having done
>> further testing I realise this is a flawed notion in and of itself, I'm
>> inadvertently expecting a queue to duplicate messages for the overlapping
>> selectors.
>>
>> Another way to describe the intending behaviour would be to have a set of
>> consumers on a topic where groups of consumers could be made where one
>> consumer in the group and only one receives a copy of the message for
>> that
>> group. Thus I can build a replicated distributed queue. The net result is
>> actually identical to using multiple queues and composite destinations.
>>
>> I think your suggestion: "So send a message to an input queue then have
>> a
>> farm of routers consuming them and routing them to the correct queue for
>> the
>> right group to process it." is the closest, when I was first thinking up
>> a
>> solution for this using composite destinations I was hoping to hide the
>> building of the composite destination from the sender. This seems like a
>> nice way to do it.
>
> Actually on reflection; Virtual Destinations should be able to use
> selectors so you can do exactly what you need. Then you'd not need to
> write any code - just configure a virtual destination.
>
> I've patched the Virtual Destination support in trunk so that you can
> configure a virtual destination as follows...
>
> <broker xmlns="http://activemq.org/config/1.0">
> <destinationInterceptors>
> <virtualDestinationInterceptor>
> <virtualDestinations>
> <compositeQueue name="MY.QUEUE">
> <forwardTo>
> <filteredDestination selector="foo = 'bar'"
> queue="Groups.A"/>
> <filteredDestination selector="x = 5" queue="Groups.B"/>
> </forwardTo>
> </compositeQueue>
> </virtualDestinations>
> </virtualDestinationInterceptor>
> </destinationInterceptors>
>
> So that if you send to queue MY.QUEUE then messages will be sent to
> queues Group.A and Group.B if they match the selectors defined for
> them.
>
> More docs on this page...
>
> http://activemq.org/site/virtual-destinations.html
>
> its all in trunk if you wanna take it for a spin?
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7564572
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message Group Overlaps and Load Balancing
Posted by James Strachan <ja...@gmail.com>.
On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
>
> Hi James,
>
> To quote: "If you just want load balancing you could use 1 queue with a
> selector
> for each group."
>
> Yes, that'd be fine, the problem is that I want an overlap. Having done
> further testing I realise this is a flawed notion in and of itself, I'm
> inadvertently expecting a queue to duplicate messages for the overlapping
> selectors.
>
> Another way to describe the intending behaviour would be to have a set of
> consumers on a topic where groups of consumers could be made where one
> consumer in the group and only one receives a copy of the message for that
> group. Thus I can build a replicated distributed queue. The net result is
> actually identical to using multiple queues and composite destinations.
>
> I think your suggestion: "So send a message to an input queue then have a
> farm of routers consuming them and routing them to the correct queue for the
> right group to process it." is the closest, when I was first thinking up a
> solution for this using composite destinations I was hoping to hide the
> building of the composite destination from the sender. This seems like a
> nice way to do it.
Actually on reflection; Virtual Destinations should be able to use
selectors so you can do exactly what you need. Then you'd not need to
write any code - just configure a virtual destination.
I've patched the Virtual Destination support in trunk so that you can
configure a virtual destination as follows...
<broker xmlns="http://activemq.org/config/1.0">
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="MY.QUEUE">
<forwardTo>
<filteredDestination selector="foo = 'bar'" queue="Groups.A"/>
<filteredDestination selector="x = 5" queue="Groups.B"/>
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
So that if you send to queue MY.QUEUE then messages will be sent to
queues Group.A and Group.B if they match the selectors defined for
them.
More docs on this page...
http://activemq.org/site/virtual-destinations.html
its all in trunk if you wanna take it for a spin?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Message Group Overlaps and Load Balancing
Posted by taxingmonk <mk...@webhanger.com>.
Hi James,
To quote: "If you just want load balancing you could use 1 queue with a
selector
for each group."
Yes, that'd be fine, the problem is that I want an overlap. Having done
further testing I realise this is a flawed notion in and of itself, I'm
inadvertently expecting a queue to duplicate messages for the overlapping
selectors.
Another way to describe the intending behaviour would be to have a set of
consumers on a topic where groups of consumers could be made where one
consumer in the group and only one receives a copy of the message for that
group. Thus I can build a replicated distributed queue. The net result is
actually identical to using multiple queues and composite destinations.
I think your suggestion: "So send a message to an input queue then have a
farm of routers consuming them and routing them to the correct queue for the
right group to process it." is the closest, when I was first thinking up a
solution for this using composite destinations I was hoping to hide the
building of the composite destination from the sender. This seems like a
nice way to do it.
Thanks James,
Max
p.s. Are you in "Norf Lundun" ATM?
James.Strachan wrote:
>
> On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
>>
>> Hi Guys,
>>
>> I've got a use case where I want to load balance a given queue but
>> paritioned accross selectors. I am convinced that somewhere between
>> composite destinations and message groups I can achieve what I'm after.
>>
>> The use case is that I want to be able to send one message which is load
>> balanced to a series of receievers which I group together somehow but at
>> the
>> same time I have multiple groups also receiving the same message.
>>
>> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group
>> 1
>> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
>> receieved by one member of Group 1, Group 2 and Group 3.
>
> You could just use 3 queues for each group - then you can browse what
> messages are available for each group & monitor the queue depth etc.
> Thats the main benefit over using selectors.
>
> Note that virtual destinations & composite destinations result in the
> message going to all of the destinations - in this use case you want
> it to only go to one of them. So selectors or a custom router is the
> best solution.
>
>
>> I thought that might be achieveable by using composite destinations but I
>> didn't want to manage a seperate queue for each group so that I could
>> drop a
>> message on the bus and have to arrive at all of them without specifying
>> send
>> (a.group1, a.group2, a.group3), etc, etc.
>>
>> Furthermore, what would be even better is if this could derived from the
>> message itself. If we consider a numberline where I can specify a range
>> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is
>> interested
>> 2-6 and group 3 is interested in 5-10. From there I would like to be able
>> to
>> say this message is for range 1
>>
>> It's almost like I want layer selector behaviour accross the top of the
>> message groups while load balancing :)
>
> Just to be clear - Message Groups are basically all about sticky load
> balancing. All queues - whether using selectors or not - implement
> load balancing.
>
> If you just want load balancing you could use 1 queue with a selector
> for each group. Using message groups on top of this should give you
> sticky load balancing if you need it.
>
>
>> Ultimatly this allows me to spawn
>> more consumers if I start to realise that the processes that are handling
>> a
>> particular range have a large data density, the key thing is that
>> multiple
>> groups can return responses to the one message, ranging specifying
>> partitions of data, which may or may not overlap. This I fear may be a
>> bridge too far.
>>
>> It should be noted that none of these messages is transactionally
>> important,
>> nor do I need the total guarentee that two or more consumers in the same
>> group receieve the same message, although that shouldn't be the common
>> case.
>
> You could write a custom Broker interceptor to implement whatever
> routing policy you wish - using the message headers / body to route to
> a different queue based on some rules...
> http://incubator.apache.org/activemq/interceptors.html
>
> Though its a bit more MOM-like to do these kinds of routing policies
> using a consumer which does the routing for you - which can then be as
> simple or complex as you wish. So send a message to an input queue
> then have a farm of routers consuming them and routing them to the
> correct queue for the right group to process it.
>
> Am not sure if you need sticky load balancing from your mail - so
> maybe just 1 queue with selectors would do the trick?
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7474172
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Message Group Overlaps and Load Balancing
Posted by James Strachan <ja...@gmail.com>.
On 11/21/06, taxingmonk <mk...@webhanger.com> wrote:
>
> Hi Guys,
>
> I've got a use case where I want to load balance a given queue but
> paritioned accross selectors. I am convinced that somewhere between
> composite destinations and message groups I can achieve what I'm after.
>
> The use case is that I want to be able to send one message which is load
> balanced to a series of receievers which I group together somehow but at the
> same time I have multiple groups also receiving the same message.
>
> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group 1
> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be
> receieved by one member of Group 1, Group 2 and Group 3.
You could just use 3 queues for each group - then you can browse what
messages are available for each group & monitor the queue depth etc.
Thats the main benefit over using selectors.
Note that virtual destinations & composite destinations result in the
message going to all of the destinations - in this use case you want
it to only go to one of them. So selectors or a custom router is the
best solution.
> I thought that might be achieveable by using composite destinations but I
> didn't want to manage a seperate queue for each group so that I could drop a
> message on the bus and have to arrive at all of them without specifying send
> (a.group1, a.group2, a.group3), etc, etc.
>
> Furthermore, what would be even better is if this could derived from the
> message itself. If we consider a numberline where I can specify a range
> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is interested
> 2-6 and group 3 is interested in 5-10. From there I would like to be able to
> say this message is for range 1
>
> It's almost like I want layer selector behaviour accross the top of the
> message groups while load balancing :)
Just to be clear - Message Groups are basically all about sticky load
balancing. All queues - whether using selectors or not - implement
load balancing.
If you just want load balancing you could use 1 queue with a selector
for each group. Using message groups on top of this should give you
sticky load balancing if you need it.
> Ultimatly this allows me to spawn
> more consumers if I start to realise that the processes that are handling a
> particular range have a large data density, the key thing is that multiple
> groups can return responses to the one message, ranging specifying
> partitions of data, which may or may not overlap. This I fear may be a
> bridge too far.
>
> It should be noted that none of these messages is transactionally important,
> nor do I need the total guarentee that two or more consumers in the same
> group receieve the same message, although that shouldn't be the common case.
You could write a custom Broker interceptor to implement whatever
routing policy you wish - using the message headers / body to route to
a different queue based on some rules...
http://incubator.apache.org/activemq/interceptors.html
Though its a bit more MOM-like to do these kinds of routing policies
using a consumer which does the routing for you - which can then be as
simple or complex as you wish. So send a message to an input queue
then have a farm of routers consuming them and routing them to the
correct queue for the right group to process it.
Am not sure if you need sticky load balancing from your mail - so
maybe just 1 queue with selectors would do the trick?
--
James
-------
http://radio.weblogs.com/0112098/