You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by apinke <ap...@gmail.com> on 2006/09/11 15:58:02 UTC
Load Balancing with Single Consumer
We are trying to achieve load balancing with single producer and single
consumer instance and multiple Brokers.
What we are trying to achive is have only one Consumer connect to one broker
and listen to a queue but for load-balancing reasons ( or if the broker goes
down ) , it should automatically connect to the next broker and consume the
messages.
Is this setup possible ?
To be more clear, can we create a "virtual" broker / queue that a producer
and consumer can use , but it is actually composed of a network of brokers ,
such that
they can share the load or provide failover capabilities.
Of course , this is not load balancing the application but only the
brokers...
thanks
Pat
--
View this message in context: http://www.nabble.com/Load-Balancing-with-Single-Consumer-tf2252613.html#a6247294
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Load Balancing with Single Consumer
Posted by James Strachan <ja...@gmail.com>.
On 9/12/06, John Heitmann <jh...@gmail.com> wrote:
>
> On Sep 11, 2006, at 11:18 PM, James Strachan wrote:
> > Currently for the latter, failover to another broker is supported.
> > However we don't yet have the former - connecting and consuming from
> > both brokers at the same time.
> >
> > There is an outstanding issue for this.
> > http://issues.apache.org/activemq/browse/AMQ-816
> >
> > we have a basic FanoutTransport that connects to multiple brokers; but
> > more work is required to add the various required semantics such as to
> > support queue consumption across many brokers and transaction support
> > etc.
> >
> > IIRC John Heitmann was working on this - any progress John?
>
> I've got a pretty much working version now. I had to pause work on it
> for a bit to take care of some other stuff, but I have time again
> now. I need to fix a couple bugs (transaction glitches and multi-
> session glitches) and get it looked at by a teammate. Once that's
> done I can post the code. I should be able to show something in a
> couple days.
Awesome! Looking forward to it :)
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancing with Single Consumer
Posted by John Heitmann <jh...@gmail.com>.
On Sep 11, 2006, at 11:18 PM, James Strachan wrote:
> Currently for the latter, failover to another broker is supported.
> However we don't yet have the former - connecting and consuming from
> both brokers at the same time.
>
> There is an outstanding issue for this.
> http://issues.apache.org/activemq/browse/AMQ-816
>
> we have a basic FanoutTransport that connects to multiple brokers; but
> more work is required to add the various required semantics such as to
> support queue consumption across many brokers and transaction support
> etc.
>
> IIRC John Heitmann was working on this - any progress John?
I've got a pretty much working version now. I had to pause work on it
for a bit to take care of some other stuff, but I have time again
now. I need to fix a couple bugs (transaction glitches and multi-
session glitches) and get it looked at by a teammate. Once that's
done I can post the code. I should be able to show something in a
couple days.
John
Re: Load Balancing with Single Consumer
Posted by James Strachan <ja...@gmail.com>.
On 9/20/06, jdrsto <jo...@metro1.com> wrote:
> James.Strachan wrote:
> >
> > On 9/11/06, apinke <ap...@gmail.com> wrote:
> >>
> >> We are trying to achieve load balancing with single producer and single
> >> consumer instance and multiple Brokers.
> >
> > Just out of interest, why? Normally a single broker can handle many
> > producers and consumers without breaking into a sweat.
> >
> >
>
> If I can offer a perspective here ...
> I've currently got a SWIFTMQ network where each "broker" is duplicated --
> not for performance, but for availability reasons. Traffic flows
> continuously, but not in duplicate, on the parallel routes. Loss of a single
> "broker" or its host does not affect the overall system and permits
> continued operation while restoring system, or when just performing
> maintenance. This does introduce the need to perform duplicate filtering for
> topics, however.
If using topics and you have a network of brokers with ActiveMQ then
the message is sent to all brokers anyway, so loosing a broker
shouldn't loose a topic message..
However for queues if you want to replicate messages for availability
but avoid duplicates (and preserve ordering etc) you need to use
Master/Slave...
http://incubator.apache.org/activemq/masterslave.html
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancing with Single Consumer
Posted by jdrsto <jo...@metro1.com>.
James.Strachan wrote:
>
> On 9/11/06, apinke <ap...@gmail.com> wrote:
>>
>> We are trying to achieve load balancing with single producer and single
>> consumer instance and multiple Brokers.
>
> Just out of interest, why? Normally a single broker can handle many
> producers and consumers without breaking into a sweat.
>
>
If I can offer a perspective here ...
I've currently got a SWIFTMQ network where each "broker" is duplicated --
not for performance, but for availability reasons. Traffic flows
continuously, but not in duplicate, on the parallel routes. Loss of a single
"broker" or its host does not affect the overall system and permits
continued operation while restoring system, or when just performing
maintenance. This does introduce the need to perform duplicate filtering for
topics, however.
--
View this message in context: http://www.nabble.com/Load-Balancing-with-Single-Consumer-tf2252613.html#a6413508
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Load Balancing with Single Consumer
Posted by James Strachan <ja...@gmail.com>.
On 9/11/06, apinke <ap...@gmail.com> wrote:
>
> We are trying to achieve load balancing with single producer and single
> consumer instance and multiple Brokers.
Just out of interest, why? Normally a single broker can handle many
producers and consumers without breaking into a sweat.
> What we are trying to achive is have only one Consumer connect to one broker
> and listen to a queue but for load-balancing reasons ( or if the broker goes
> down ) , it should automatically connect to the next broker and consume the
> messages.
> Is this setup possible ?
Currently for the latter, failover to another broker is supported.
However we don't yet have the former - connecting and consuming from
both brokers at the same time.
There is an outstanding issue for this.
http://issues.apache.org/activemq/browse/AMQ-816
we have a basic FanoutTransport that connects to multiple brokers; but
more work is required to add the various required semantics such as to
support queue consumption across many brokers and transaction support
etc.
IIRC John Heitmann was working on this - any progress John?
> To be more clear, can we create a "virtual" broker / queue that a producer
> and consumer can use , but it is actually composed of a network of brokers ,
> such that
> they can share the load or provide failover capabilities.
> Of course , this is not load balancing the application but only the
> brokers...
The other option is you create a network of brokers with
producers/consumers connecting to one of the brokers and the brokers
store-and-forwarding to other consumers on other brokers when needed.
Ultimately the fanount (or 'jedi' as we've referred to it in the past)
transport could be what you need.
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancing with Single Consumer
Posted by Adrian Co <ac...@exist.com>.
What do you mean it didn't work? Did you get any exceptions?
Could you post the broker and client output/logs?
apinke wrote:
> I did try setting up Multicast as mentioned in this post :
> http://www.nabble.com/broker-discovery-client-help-tf1651573.html#a4474136
>
> but the brokers just sat there and the producer did not send any messages at
> all !
>
>
> Broker X:
> <transportConnector uri="tcp://localhost:61616"
> discoveryUri="multicast://foo"/>
> and network connector is:
> <networkConnector uri="multicast://foo">
>
> Broker Y:
> <transportConnector uri="tcp://localhost:61617"
> discoveryUri="multicast://foo"/>
> and network connector is:
> <networkConnector uri="multicast://foo">
>
> The producer used the uri :
> discovery:multicast://foo
>
> It didnt work. Anything wrong in the config ?
>
> Thanks
> Pat
>
>
>
> Adrian Co wrote:
>
>> Do you mean having a multicast group of brokers?
>> http://www.activemq.org/site/multicast-transport-reference.html
>>
>> apinke wrote:
>>
>>> Thanks Adrian..
>>>
>>> We already went through these and did manage to set up a network..the
>>> problem was
>>> 1) the other broker was used only when there was a failover and not as a
>>> load balancer
>>> 2) the other broker needed its own set of consumers.
>>>
>>> What we wanted to try out was if the 2 brokers would load balance but
>>> there
>>> would be only 1 consumer..
>>>
>>> thanks
>>> Pat
>>>
>>>
>>> Adrian Co wrote:
>>>
>>>
>>>> Wonder if these is what you are looking for:
>>>> http://www.activemq.org/site/networks-of-brokers.html
>>>> http://www.activemq.org/site/failover-transport-reference.html
>>>> http://www.activemq.org/site/masterslave.html
>>>>
>>>> apinke wrote:
>>>>
>>>>
>>>>> We are trying to achieve load balancing with single producer and single
>>>>> consumer instance and multiple Brokers.
>>>>>
>>>>> What we are trying to achive is have only one Consumer connect to one
>>>>> broker
>>>>> and listen to a queue but for load-balancing reasons ( or if the broker
>>>>> goes
>>>>> down ) , it should automatically connect to the next broker and
>>>>> consume
>>>>> the
>>>>> messages.
>>>>> Is this setup possible ?
>>>>>
>>>>> To be more clear, can we create a "virtual" broker / queue that a
>>>>> producer
>>>>> and consumer can use , but it is actually composed of a network of
>>>>> brokers ,
>>>>> such that
>>>>> they can share the load or provide failover capabilities.
>>>>> Of course , this is not load balancing the application but only the
>>>>> brokers...
>>>>>
>>>>> thanks
>>>>> Pat
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: Load Balancing with Single Consumer
Posted by apinke <ap...@gmail.com>.
I did try setting up Multicast as mentioned in this post :
http://www.nabble.com/broker-discovery-client-help-tf1651573.html#a4474136
but the brokers just sat there and the producer did not send any messages at
all !
Broker X:
<transportConnector uri="tcp://localhost:61616"
discoveryUri="multicast://foo"/>
and network connector is:
<networkConnector uri="multicast://foo">
Broker Y:
<transportConnector uri="tcp://localhost:61617"
discoveryUri="multicast://foo"/>
and network connector is:
<networkConnector uri="multicast://foo">
The producer used the uri :
discovery:multicast://foo
It didnt work. Anything wrong in the config ?
Thanks
Pat
Adrian Co wrote:
>
> Do you mean having a multicast group of brokers?
> http://www.activemq.org/site/multicast-transport-reference.html
>
> apinke wrote:
>> Thanks Adrian..
>>
>> We already went through these and did manage to set up a network..the
>> problem was
>> 1) the other broker was used only when there was a failover and not as a
>> load balancer
>> 2) the other broker needed its own set of consumers.
>>
>> What we wanted to try out was if the 2 brokers would load balance but
>> there
>> would be only 1 consumer..
>>
>> thanks
>> Pat
>>
>>
>> Adrian Co wrote:
>>
>>> Wonder if these is what you are looking for:
>>> http://www.activemq.org/site/networks-of-brokers.html
>>> http://www.activemq.org/site/failover-transport-reference.html
>>> http://www.activemq.org/site/masterslave.html
>>>
>>> apinke wrote:
>>>
>>>> We are trying to achieve load balancing with single producer and single
>>>> consumer instance and multiple Brokers.
>>>>
>>>> What we are trying to achive is have only one Consumer connect to one
>>>> broker
>>>> and listen to a queue but for load-balancing reasons ( or if the broker
>>>> goes
>>>> down ) , it should automatically connect to the next broker and
>>>> consume
>>>> the
>>>> messages.
>>>> Is this setup possible ?
>>>>
>>>> To be more clear, can we create a "virtual" broker / queue that a
>>>> producer
>>>> and consumer can use , but it is actually composed of a network of
>>>> brokers ,
>>>> such that
>>>> they can share the load or provide failover capabilities.
>>>> Of course , this is not load balancing the application but only the
>>>> brokers...
>>>>
>>>> thanks
>>>> Pat
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
--
View this message in context: http://www.nabble.com/Load-Balancing-with-Single-Consumer-tf2252613.html#a6263142
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Load Balancing with Single Consumer
Posted by apinke <ap...@gmail.com>.
I did try setting up Multicast as mentioned in this post :
http://www.nabble.com/broker-discovery-client-help-tf1651573.html#a4474136
but the brokers just sat there and the producer did not send any messages at
all !
Broker X:
<transportConnector uri="tcp://localhost:61616"
discoveryUri="multicast://foo"/>
and network connector is:
<networkConnector uri="multicast://foo">
Broker Y:
Broker X:
<transportConnector uri="tcp://localhost:61617"
discoveryUri="multicast://foo"/>
and network connector is:
<networkConnector uri="multicast://foo">
The producer used the uri :
discovery:multicast://foo
It didnt work. Anything wrong in the config ?
Thanks
Pat
Adrian Co wrote:
>
> Do you mean having a multicast group of brokers?
> http://www.activemq.org/site/multicast-transport-reference.html
>
> apinke wrote:
>> Thanks Adrian..
>>
>> We already went through these and did manage to set up a network..the
>> problem was
>> 1) the other broker was used only when there was a failover and not as a
>> load balancer
>> 2) the other broker needed its own set of consumers.
>>
>> What we wanted to try out was if the 2 brokers would load balance but
>> there
>> would be only 1 consumer..
>>
>> thanks
>> Pat
>>
>>
>> Adrian Co wrote:
>>
>>> Wonder if these is what you are looking for:
>>> http://www.activemq.org/site/networks-of-brokers.html
>>> http://www.activemq.org/site/failover-transport-reference.html
>>> http://www.activemq.org/site/masterslave.html
>>>
>>> apinke wrote:
>>>
>>>> We are trying to achieve load balancing with single producer and single
>>>> consumer instance and multiple Brokers.
>>>>
>>>> What we are trying to achive is have only one Consumer connect to one
>>>> broker
>>>> and listen to a queue but for load-balancing reasons ( or if the broker
>>>> goes
>>>> down ) , it should automatically connect to the next broker and
>>>> consume
>>>> the
>>>> messages.
>>>> Is this setup possible ?
>>>>
>>>> To be more clear, can we create a "virtual" broker / queue that a
>>>> producer
>>>> and consumer can use , but it is actually composed of a network of
>>>> brokers ,
>>>> such that
>>>> they can share the load or provide failover capabilities.
>>>> Of course , this is not load balancing the application but only the
>>>> brokers...
>>>>
>>>> thanks
>>>> Pat
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
--
View this message in context: http://www.nabble.com/Load-Balancing-with-Single-Consumer-tf2252613.html#a6263142
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Load Balancing with Single Consumer
Posted by Adrian Co <ac...@exist.com>.
Do you mean having a multicast group of brokers?
http://www.activemq.org/site/multicast-transport-reference.html
apinke wrote:
> Thanks Adrian..
>
> We already went through these and did manage to set up a network..the
> problem was
> 1) the other broker was used only when there was a failover and not as a
> load balancer
> 2) the other broker needed its own set of consumers.
>
> What we wanted to try out was if the 2 brokers would load balance but there
> would be only 1 consumer..
>
> thanks
> Pat
>
>
> Adrian Co wrote:
>
>> Wonder if these is what you are looking for:
>> http://www.activemq.org/site/networks-of-brokers.html
>> http://www.activemq.org/site/failover-transport-reference.html
>> http://www.activemq.org/site/masterslave.html
>>
>> apinke wrote:
>>
>>> We are trying to achieve load balancing with single producer and single
>>> consumer instance and multiple Brokers.
>>>
>>> What we are trying to achive is have only one Consumer connect to one
>>> broker
>>> and listen to a queue but for load-balancing reasons ( or if the broker
>>> goes
>>> down ) , it should automatically connect to the next broker and consume
>>> the
>>> messages.
>>> Is this setup possible ?
>>>
>>> To be more clear, can we create a "virtual" broker / queue that a
>>> producer
>>> and consumer can use , but it is actually composed of a network of
>>> brokers ,
>>> such that
>>> they can share the load or provide failover capabilities.
>>> Of course , this is not load balancing the application but only the
>>> brokers...
>>>
>>> thanks
>>> Pat
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Load Balancing with Single Consumer
Posted by apinke <ap...@gmail.com>.
Thanks Adrian..
We already went through these and did manage to set up a network..the
problem was
1) the other broker was used only when there was a failover and not as a
load balancer
2) the other broker needed its own set of consumers.
What we wanted to try out was if the 2 brokers would load balance but there
would be only 1 consumer..
thanks
Pat
Adrian Co wrote:
>
> Wonder if these is what you are looking for:
> http://www.activemq.org/site/networks-of-brokers.html
> http://www.activemq.org/site/failover-transport-reference.html
> http://www.activemq.org/site/masterslave.html
>
> apinke wrote:
>> We are trying to achieve load balancing with single producer and single
>> consumer instance and multiple Brokers.
>>
>> What we are trying to achive is have only one Consumer connect to one
>> broker
>> and listen to a queue but for load-balancing reasons ( or if the broker
>> goes
>> down ) , it should automatically connect to the next broker and consume
>> the
>> messages.
>> Is this setup possible ?
>>
>> To be more clear, can we create a "virtual" broker / queue that a
>> producer
>> and consumer can use , but it is actually composed of a network of
>> brokers ,
>> such that
>> they can share the load or provide failover capabilities.
>> Of course , this is not load balancing the application but only the
>> brokers...
>>
>> thanks
>> Pat
>>
>>
>>
>
>
>
--
View this message in context: http://www.nabble.com/Load-Balancing-with-Single-Consumer-tf2252613.html#a6259485
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Load Balancing with Single Consumer
Posted by Adrian Co <ac...@exist.com>.
Wonder if these is what you are looking for:
http://www.activemq.org/site/networks-of-brokers.html
http://www.activemq.org/site/failover-transport-reference.html
http://www.activemq.org/site/masterslave.html
apinke wrote:
> We are trying to achieve load balancing with single producer and single
> consumer instance and multiple Brokers.
>
> What we are trying to achive is have only one Consumer connect to one broker
> and listen to a queue but for load-balancing reasons ( or if the broker goes
> down ) , it should automatically connect to the next broker and consume the
> messages.
> Is this setup possible ?
>
> To be more clear, can we create a "virtual" broker / queue that a producer
> and consumer can use , but it is actually composed of a network of brokers ,
> such that
> they can share the load or provide failover capabilities.
> Of course , this is not load balancing the application but only the
> brokers...
>
> thanks
> Pat
>
>
>