You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Sanjiv Jivan <sa...@gmail.com> on 2006/07/06 16:41:35 UTC
load balancing across network of brokers
This is related to the discussions in the "detaching message group affinity"
thread (
http://www.nabble.com/detaching-message-group-affinity-tf1801568.html#a4910294
)
Here's my deployment configuration. I have multiple "servers", each of which
embed a activemq broker. The server app is the only recipient of its
embedded broker inbound messages. The embedded brokers are configured as a
"Network of brokers" ie a peer topology.
Is load balancing supported in such a deployment? For example when a client
sends multiple messages to the network of brokers ( ie broker URL like
"discovery:multicast://myapp"), will the messages be load balanced across
all the brokers in the peer network? If so, I'm not seeing this behavior. In
a configuration with 2 servers (w/ embedded brokers), when a client sends 30
messages, 28 of them go to one broker and only a couple of them go to the
other broker.
In another thread James mentioned that "there is no global cross-broker load
balancing.". So I'm guessing this sort of load balancing is not supported.
Can support for this be added? The client already is aware of the list of
brokers that are part of the peer network so a simple round robin style load
balancing should not be too complicated. Additionally if message groups are
involved, a check could be made at the client to see if a message with that
message group already had a assigned broker and if so send it to that one,
else pick another broker from the list of brokers and assign the message
group to this broker.
I have described my use case along with code here :
http://issues.apache.org/activemq/browse/AMQ-762
I would like to know if this is something you'd consider else I'll have to
look for alternate solutions like using a distributed hash map and add the
logic in the client to carry out such cross broker load balancing which also
respects message groups. I'd rather not go down this path since ActiveMQ
already has a distributed map of brokers that the client is aware of and it
almost supports what I require.
Thanks,
Sanjiv
---------- Forwarded message ----------
From: James Strachan <ja...@gmail.com>
Date: Jun 19, 2006 10:49 AM
Subject: Re: detaching message group affinity
To: activemq-users@geronimo.apache.org
On 6/17/06, Sanjiv Jivan <sa...@gmail.com> wrote:
> The Message Group functionality of Active MQ 4.0 is really neat. I have a
> couple of questions regarding this
>
> 1. Is this feature fully supported/tested with the P2P discovery style
> broker topology? Are they any test cases or samples I can try. I looked
for
> them in the source but wasn't able to locate them. And when I tried
running
> such a scenario, I did not see a proper load balancing (even round robin
> style) of message groups across available brokers. When messages belonging
> to different message groups are sent, oftentimes only one or a small
> percentage of running brokers are chosen as recipients. As a result I end
up
> with some brokers that are overloaded while others are sitting idle.
Generally Message Groups only apply to the broker which you are using;
there is no global cross-broker load balancing. So its intended when
using a traditional client-broker topology rather than a peer based
network.
Though I'm surprised that you are finding on a single broker it is not
load balancing requests across multiple brokers. Are you sure there
are suitable consumers on those brokers?
Re: load balancing across network of brokers
Posted by Sanjiv Jivan <sa...@gmail.com>.
I'm not seeing load balancing behavior across brokers even without message
groups. I'll create a testcase for this. Let me know if there's already a
test case for loadbalancing which I might be able to tweak for a network of
brokers configuration.
Thanks,
Sanjiv
On 7/24/06, Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> Load balancing should work across brokers. The issue of message groups
> working across multiple brokers is a different one. Want to tackle each
> one separately? Also, submitting junit tests showing the problem is the
> best way for folks to get involved. Perhaps some out there figures out
> how
> to fix your test cases and submits a patch.
>
> Regards,
> Hiram
>
> On 7/22/06, Sanjiv Jivan <sa...@gmail.com> wrote:
> >
> > James, Hiram any thoughts on this?
> >
> > ---------- Forwarded message ----------
> > From: Sanjiv Jivan <sa...@gmail.com>
> > Date: Jul 6, 2006 10:41 AM
> > Subject: load balancing across network of brokers
> > To: activemq-users@geronimo.apache.org
> >
> > This is related to the discussions in the "detaching message group
> > affinity" thread
> > (
> >
> http://www.nabble.com/detaching-message-group-affinity-tf1801568.html#a4910294
> > )
> >
> > Here's my deployment configuration. I have multiple "servers", each of
> > which
> > embed a activemq broker. The server app is the only recipient of its
> > embedded broker inbound messages. The embedded brokers are configured as
> a
> > "Network of brokers" ie a peer topology.
> >
> > Is load balancing supported in such a deployment? For example when a
> > client
> > sends multiple messages to the network of brokers ( ie broker URL like
> > "discovery:multicast://myapp"), will the messages be load balanced
> across
> > all the brokers in the peer network? If so, I'm not seeing this
> behavior.
> > In
> > a configuration with 2 servers (w/ embedded brokers), when a client
> sends
> > 30
> > messages, 28 of them go to one broker and only a couple of them go to
> the
> > other broker.
> >
> > In another thread James mentioned that "there is no global cross-broker
> > load
> > balancing.". So I'm guessing this sort of load balancing is not
> supported.
> > Can support for this be added? The client already is aware of the list
> of
> > brokers that are part of the peer network so a simple round robin style
> > load
> > balancing should not be too complicated. Additionally if message groups
> > are
> > involved, a check could be made at the client to see if a message with
> > that
> > message group already had a assigned broker and if so send it to that
> one,
> > else pick another broker from the list of brokers and assign the message
> > group to this broker.
> >
> > I have described my use case along with code here :
> > http://issues.apache.org/activemq/browse/AMQ-762
> >
> > I would like to know if this is something you'd consider else I'll have
> to
> > look for alternate solutions like using a distributed hash map and add
> the
> > logic in the client to carry out such cross broker load balancing which
> > also
> > respects message groups. I'd rather not go down this path since ActiveMQ
> > already has a distributed map of brokers that the client is aware of and
> > it
> > almost supports what I require.
> >
> > Thanks,
> > Sanjiv
> >
> > ---------- Forwarded message ----------
> > From: James Strachan <ja...@gmail.com>
> > Date: Jun 19, 2006 10:49 AM
> > Subject: Re: detaching message group affinity
> > To: activemq-users@geronimo.apache.org
> >
> > On 6/17/06, Sanjiv Jivan < sanjiv.jivan@gmail.com> wrote:
> > > The Message Group functionality of Active MQ 4.0 is really neat. I
> have
> > a
> > > couple of questions regarding this
> > >
> > > 1. Is this feature fully supported/tested with the P2P discovery style
> > > broker topology? Are they any test cases or samples I can try. I
> looked
> > for
> > > them in the source but wasn't able to locate them. And when I tried
> > running
> > > such a scenario, I did not see a proper load balancing (even round
> robin
> > > style) of message groups across available brokers. When messages
> > belonging
> > > to different message groups are sent, oftentimes only one or a small
> > > percentage of running brokers are chosen as recipients. As a result I
> > end
> > up
> > > with some brokers that are overloaded while others are sitting idle.
> >
> > Generally Message Groups only apply to the broker which you are using;
> > there is no global cross-broker load balancing. So its intended when
> > using a traditional client-broker topology rather than a peer based
> > network.
> >
> > Though I'm surprised that you are finding on a single broker it is not
> > load balancing requests across multiple brokers. Are you sure there
> > are suitable consumers on those brokers?
> >
> >
>
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>
>
Re: load balancing across network of brokers
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Load balancing should work across brokers. The issue of message groups
working across multiple brokers is a different one. Want to tackle each
one separately? Also, submitting junit tests showing the problem is the
best way for folks to get involved. Perhaps some out there figures out how
to fix your test cases and submits a patch.
Regards,
Hiram
On 7/22/06, Sanjiv Jivan <sa...@gmail.com> wrote:
>
> James, Hiram any thoughts on this?
>
> ---------- Forwarded message ----------
> From: Sanjiv Jivan <sa...@gmail.com>
> Date: Jul 6, 2006 10:41 AM
> Subject: load balancing across network of brokers
> To: activemq-users@geronimo.apache.org
>
> This is related to the discussions in the "detaching message group
> affinity" thread
> (
> http://www.nabble.com/detaching-message-group-affinity-tf1801568.html#a4910294
> )
>
> Here's my deployment configuration. I have multiple "servers", each of
> which
> embed a activemq broker. The server app is the only recipient of its
> embedded broker inbound messages. The embedded brokers are configured as a
> "Network of brokers" ie a peer topology.
>
> Is load balancing supported in such a deployment? For example when a
> client
> sends multiple messages to the network of brokers ( ie broker URL like
> "discovery:multicast://myapp"), will the messages be load balanced across
> all the brokers in the peer network? If so, I'm not seeing this behavior.
> In
> a configuration with 2 servers (w/ embedded brokers), when a client sends
> 30
> messages, 28 of them go to one broker and only a couple of them go to the
> other broker.
>
> In another thread James mentioned that "there is no global cross-broker
> load
> balancing.". So I'm guessing this sort of load balancing is not supported.
> Can support for this be added? The client already is aware of the list of
> brokers that are part of the peer network so a simple round robin style
> load
> balancing should not be too complicated. Additionally if message groups
> are
> involved, a check could be made at the client to see if a message with
> that
> message group already had a assigned broker and if so send it to that one,
> else pick another broker from the list of brokers and assign the message
> group to this broker.
>
> I have described my use case along with code here :
> http://issues.apache.org/activemq/browse/AMQ-762
>
> I would like to know if this is something you'd consider else I'll have to
> look for alternate solutions like using a distributed hash map and add the
> logic in the client to carry out such cross broker load balancing which
> also
> respects message groups. I'd rather not go down this path since ActiveMQ
> already has a distributed map of brokers that the client is aware of and
> it
> almost supports what I require.
>
> Thanks,
> Sanjiv
>
> ---------- Forwarded message ----------
> From: James Strachan <ja...@gmail.com>
> Date: Jun 19, 2006 10:49 AM
> Subject: Re: detaching message group affinity
> To: activemq-users@geronimo.apache.org
>
> On 6/17/06, Sanjiv Jivan < sanjiv.jivan@gmail.com> wrote:
> > The Message Group functionality of Active MQ 4.0 is really neat. I have
> a
> > couple of questions regarding this
> >
> > 1. Is this feature fully supported/tested with the P2P discovery style
> > broker topology? Are they any test cases or samples I can try. I looked
> for
> > them in the source but wasn't able to locate them. And when I tried
> running
> > such a scenario, I did not see a proper load balancing (even round robin
> > style) of message groups across available brokers. When messages
> belonging
> > to different message groups are sent, oftentimes only one or a small
> > percentage of running brokers are chosen as recipients. As a result I
> end
> up
> > with some brokers that are overloaded while others are sitting idle.
>
> Generally Message Groups only apply to the broker which you are using;
> there is no global cross-broker load balancing. So its intended when
> using a traditional client-broker topology rather than a peer based
> network.
>
> Though I'm surprised that you are finding on a single broker it is not
> load balancing requests across multiple brokers. Are you sure there
> are suitable consumers on those brokers?
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Fwd: load balancing across network of brokers
Posted by Sanjiv Jivan <sa...@gmail.com>.
James, Hiram any thoughts on this?
---------- Forwarded message ----------
From: Sanjiv Jivan <sa...@gmail.com>
Date: Jul 6, 2006 10:41 AM
Subject: load balancing across network of brokers
To: activemq-users@geronimo.apache.org
This is related to the discussions in the "detaching message group
affinity" thread
(http://www.nabble.com/detaching-message-group-affinity-tf1801568.html#a4910294
)
Here's my deployment configuration. I have multiple "servers", each of which
embed a activemq broker. The server app is the only recipient of its
embedded broker inbound messages. The embedded brokers are configured as a
"Network of brokers" ie a peer topology.
Is load balancing supported in such a deployment? For example when a client
sends multiple messages to the network of brokers ( ie broker URL like
"discovery:multicast://myapp"), will the messages be load balanced across
all the brokers in the peer network? If so, I'm not seeing this behavior. In
a configuration with 2 servers (w/ embedded brokers), when a client sends 30
messages, 28 of them go to one broker and only a couple of them go to the
other broker.
In another thread James mentioned that "there is no global cross-broker load
balancing.". So I'm guessing this sort of load balancing is not supported.
Can support for this be added? The client already is aware of the list of
brokers that are part of the peer network so a simple round robin style load
balancing should not be too complicated. Additionally if message groups are
involved, a check could be made at the client to see if a message with that
message group already had a assigned broker and if so send it to that one,
else pick another broker from the list of brokers and assign the message
group to this broker.
I have described my use case along with code here :
http://issues.apache.org/activemq/browse/AMQ-762
I would like to know if this is something you'd consider else I'll have to
look for alternate solutions like using a distributed hash map and add the
logic in the client to carry out such cross broker load balancing which also
respects message groups. I'd rather not go down this path since ActiveMQ
already has a distributed map of brokers that the client is aware of and it
almost supports what I require.
Thanks,
Sanjiv
---------- Forwarded message ----------
From: James Strachan <ja...@gmail.com>
Date: Jun 19, 2006 10:49 AM
Subject: Re: detaching message group affinity
To: activemq-users@geronimo.apache.org
On 6/17/06, Sanjiv Jivan < sanjiv.jivan@gmail.com> wrote:
> The Message Group functionality of Active MQ 4.0 is really neat. I have a
> couple of questions regarding this
>
> 1. Is this feature fully supported/tested with the P2P discovery style
> broker topology? Are they any test cases or samples I can try. I looked
for
> them in the source but wasn't able to locate them. And when I tried
running
> such a scenario, I did not see a proper load balancing (even round robin
> style) of message groups across available brokers. When messages belonging
> to different message groups are sent, oftentimes only one or a small
> percentage of running brokers are chosen as recipients. As a result I end
up
> with some brokers that are overloaded while others are sitting idle.
Generally Message Groups only apply to the broker which you are using;
there is no global cross-broker load balancing. So its intended when
using a traditional client-broker topology rather than a peer based
network.
Though I'm surprised that you are finding on a single broker it is not
load balancing requests across multiple brokers. Are you sure there
are suitable consumers on those brokers?