You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ttmdev <jo...@ttmsolutions.com> on 2008/03/01 00:01:39 UTC
Re: Single consumer for multiple brokers?
Yes, I got similar results in my consumer testing. What version of AMQ are
you using?
Your only recourse may be to multi thread your consumer and have it create a
connection to each of the brokers.
Joe
Stepan Koltsov wrote:
>
> I could not make make fanout transport work for consuming. I configured
> transport as:
>
> fanout:(static:(tcp://localhost:12346,tcp://localhost:12345))
>
> and messages were consumed only from one broker, and there were a lot of
> errors in log.
>
> S.
>
>
> ttmdev wrote:
>>
>> There is the composite 'fanout' transport, which allows you to replicate
>> a command across multiple brokers. However, when I last tested fanout, it
>> worked well for producing, but not so well for consuming.
>>
>> http://activemq.apache.org/fanout-transport-reference.html
>>
>> There has also been some talk of a composite 'jedi' transport that would
>> allow your consumer to fanout receives across multiple brokers, but I
>> don't believe this has been implemented.
>>
>> Joe
>> www.ttmsolutions.com
>>
>>
>> Stepan Koltsov wrote:
>>>
>>> Hi,
>>>
>>> please, help.
>>>
>>> Is it possible to connect single consumer to several brokers?
>>>
>>> There are several hosts with producers, each host has own broker.
>>>
>>> There are several hosts with consumers. Each consumer should consume
>>> messages from all producers.
>>>
>>> How is it possible to configure ActiveMQ for such configuration? I don't
>>> want to create one more broker that collects messages from all brokers.
>>>
>>> I haven't found transport like
>>> "collect:(tcp://host1:12345,tcp://host2:12345)".
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836s2354p15769003.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Single consumer for multiple brokers?
Posted by Rob Davies <ra...@gmail.com>.
http://activemq.apache.org/fanout-transport-reference.html - is the
place to look to get started
On 29 Jan 2009, at 17:00, James Strachan wrote:
> The fanout transport kinda has most of the implementation code to do
> this; it just needs to be hacked a little so that consume start/stop
> commands are sent to multiple brokers and transactions/acks are
> correctly sent to the broker that sent the original message. There
> might be some fun and games with transactions in there mind; but for
> simple-ish use cases it shouldn't be too hard to hack the
> FanoutTransport if anyone fancies having a go
>
>
> 2009/1/29 Jim Lloyd <jl...@silvertailsystems.com>:
>> I'm interested in a variation of this for load balancing. Assume
>> the volume
>> of data coming from publishers is very large, and you need N
>> brokers (think
>> N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so
>> that if any
>> 1 or 2 brokers die, there is sufficient capacity to continue
>> handling all of
>> the traffic (after affected publishers reconnect). The brokers
>> might all be
>> behind a hardware load balancer (e.g. F5 or NetScaler) so that all
>> publishers connect via a virtual IP.
>>
>> Now, from the consumer side, we also have many consumers. Each
>> consumer only
>> subscribes to a subset of the available topics, and we arrange via
>> design of
>> our topics such that one consumer can always keep up with the
>> volume of data
>> published on one topic. But the consumer must connect to every
>> broker.
>>
>> So, we need a fan-in variation of a failover transport that
>> connects to
>> every broker and actively consumes data from all brokers. If any
>> broker
>> disconnects, the consumer would use the exponential backoff
>> reconnect logic
>> to reconnect when the broker becomes available.
>>
>> So, the backup=true option isn't helpful. Instead, we want
>> something like
>> "fan_in=true".
>>
>> It's not hard to build this kind of fan in logic on top of the
>> failover
>> transport, but it would be cool if the failover transport was
>> capable of
>> doing fan in directly.
>>
>> On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <ga...@gmail.com>
>> wrote:
>>
>>> have you looked at the failover transport? see:
>>> http://activemq.apache.org/failover-transport-reference.html
>>> with the backup=true option, a connection to all listed brokers will
>>> be created so that they are in hot standby in
>>> the event that the first connection is dropped. In this way,
>>> failover
>>> can be very fast.
>>>
>>> 2009/1/29 kaykay <ka...@gmail.com>:
>>>>
>>>> This thread is old but just curious if there has been a recent
>>>> update of
>>> this
>>>> w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is
>>>> similar
>>> where
>>>> a consumer listens to multiple brokers (as a failover redundancy
>>>> issue
>>>> instead of listening to a single broker).
>>>>
>>>>
>>>> Stepan Koltsov wrote:
>>>>>
>>>>> I'm playing with 5.0.
>>>>>
>>>>> How do you think, is it hard to write new Transport that
>>>>> consumes from
>>>>> multiple brokers?
>>>>>
>>>>> S.
>>>>>
>>>>>
>>>>> ttmdev wrote:
>>>>>>
>>>>>> Yes, I got similar results in my consumer testing. What version
>>>>>> of AMQ
>>>>>> are you using?
>>>>>>
>>>>>> Your only recourse may be to multi thread your consumer and
>>>>>> have it
>>>>>> create a connection to each of the brokers.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source SOA
>>> http://FUSESource.com
>>>
>>
>
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
Re: Single consumer for multiple brokers?
Posted by James Strachan <ja...@gmail.com>.
The fanout transport kinda has most of the implementation code to do
this; it just needs to be hacked a little so that consume start/stop
commands are sent to multiple brokers and transactions/acks are
correctly sent to the broker that sent the original message. There
might be some fun and games with transactions in there mind; but for
simple-ish use cases it shouldn't be too hard to hack the
FanoutTransport if anyone fancies having a go
2009/1/29 Jim Lloyd <jl...@silvertailsystems.com>:
> I'm interested in a variation of this for load balancing. Assume the volume
> of data coming from publishers is very large, and you need N brokers (think
> N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so that if any
> 1 or 2 brokers die, there is sufficient capacity to continue handling all of
> the traffic (after affected publishers reconnect). The brokers might all be
> behind a hardware load balancer (e.g. F5 or NetScaler) so that all
> publishers connect via a virtual IP.
>
> Now, from the consumer side, we also have many consumers. Each consumer only
> subscribes to a subset of the available topics, and we arrange via design of
> our topics such that one consumer can always keep up with the volume of data
> published on one topic. But the consumer must connect to every broker.
>
> So, we need a fan-in variation of a failover transport that connects to
> every broker and actively consumes data from all brokers. If any broker
> disconnects, the consumer would use the exponential backoff reconnect logic
> to reconnect when the broker becomes available.
>
> So, the backup=true option isn't helpful. Instead, we want something like
> "fan_in=true".
>
> It's not hard to build this kind of fan in logic on top of the failover
> transport, but it would be cool if the failover transport was capable of
> doing fan in directly.
>
> On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <ga...@gmail.com> wrote:
>
>> have you looked at the failover transport? see:
>> http://activemq.apache.org/failover-transport-reference.html
>> with the backup=true option, a connection to all listed brokers will
>> be created so that they are in hot standby in
>> the event that the first connection is dropped. In this way, failover
>> can be very fast.
>>
>> 2009/1/29 kaykay <ka...@gmail.com>:
>> >
>> > This thread is old but just curious if there has been a recent update of
>> this
>> > w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar
>> where
>> > a consumer listens to multiple brokers (as a failover redundancy issue
>> > instead of listening to a single broker).
>> >
>> >
>> > Stepan Koltsov wrote:
>> >>
>> >> I'm playing with 5.0.
>> >>
>> >> How do you think, is it hard to write new Transport that consumes from
>> >> multiple brokers?
>> >>
>> >> S.
>> >>
>> >>
>> >> ttmdev wrote:
>> >>>
>> >>> Yes, I got similar results in my consumer testing. What version of AMQ
>> >>> are you using?
>> >>>
>> >>> Your only recourse may be to multi thread your consumer and have it
>> >>> create a connection to each of the brokers.
>> >>>
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://fusesource.com/
Re: Single consumer for multiple brokers?
Posted by Jim Lloyd <jl...@silvertailsystems.com>.
I'm interested in a variation of this for load balancing. Assume the volume
of data coming from publishers is very large, and you need N brokers (think
N is 10 or more). Furthermore, you want N+1 or N+2 redundancy so that if any
1 or 2 brokers die, there is sufficient capacity to continue handling all of
the traffic (after affected publishers reconnect). The brokers might all be
behind a hardware load balancer (e.g. F5 or NetScaler) so that all
publishers connect via a virtual IP.
Now, from the consumer side, we also have many consumers. Each consumer only
subscribes to a subset of the available topics, and we arrange via design of
our topics such that one consumer can always keep up with the volume of data
published on one topic. But the consumer must connect to every broker.
So, we need a fan-in variation of a failover transport that connects to
every broker and actively consumes data from all brokers. If any broker
disconnects, the consumer would use the exponential backoff reconnect logic
to reconnect when the broker becomes available.
So, the backup=true option isn't helpful. Instead, we want something like
"fan_in=true".
It's not hard to build this kind of fan in logic on top of the failover
transport, but it would be cool if the failover transport was capable of
doing fan in directly.
On Thu, Jan 29, 2009 at 8:13 AM, Gary Tully <ga...@gmail.com> wrote:
> have you looked at the failover transport? see:
> http://activemq.apache.org/failover-transport-reference.html
> with the backup=true option, a connection to all listed brokers will
> be created so that they are in hot standby in
> the event that the first connection is dropped. In this way, failover
> can be very fast.
>
> 2009/1/29 kaykay <ka...@gmail.com>:
> >
> > This thread is old but just curious if there has been a recent update of
> this
> > w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar
> where
> > a consumer listens to multiple brokers (as a failover redundancy issue
> > instead of listening to a single broker).
> >
> >
> > Stepan Koltsov wrote:
> >>
> >> I'm playing with 5.0.
> >>
> >> How do you think, is it hard to write new Transport that consumes from
> >> multiple brokers?
> >>
> >> S.
> >>
> >>
> >> ttmdev wrote:
> >>>
> >>> Yes, I got similar results in my consumer testing. What version of AMQ
> >>> are you using?
> >>>
> >>> Your only recourse may be to multi thread your consumer and have it
> >>> create a connection to each of the brokers.
> >>>
> >>
> >>
> >
> > --
> > View this message in context:
> http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
Re: Single consumer for multiple brokers?
Posted by Gary Tully <ga...@gmail.com>.
have you looked at the failover transport? see:
http://activemq.apache.org/failover-transport-reference.html
with the backup=true option, a connection to all listed brokers will
be created so that they are in hot standby in
the event that the first connection is dropped. In this way, failover
can be very fast.
2009/1/29 kaykay <ka...@gmail.com>:
>
> This thread is old but just curious if there has been a recent update of this
> w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar where
> a consumer listens to multiple brokers (as a failover redundancy issue
> instead of listening to a single broker).
>
>
> Stepan Koltsov wrote:
>>
>> I'm playing with 5.0.
>>
>> How do you think, is it hard to write new Transport that consumes from
>> multiple brokers?
>>
>> S.
>>
>>
>> ttmdev wrote:
>>>
>>> Yes, I got similar results in my consumer testing. What version of AMQ
>>> are you using?
>>>
>>> Your only recourse may be to multi thread your consumer and have it
>>> create a connection to each of the brokers.
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Single consumer for multiple brokers?
Posted by kaykay <ka...@gmail.com>.
This thread is old but just curious if there has been a recent update of this
w.r.t ActiveMQ 5.2 . The problem that I am trying to solve is similar where
a consumer listens to multiple brokers (as a failover redundancy issue
instead of listening to a single broker).
Stepan Koltsov wrote:
>
> I'm playing with 5.0.
>
> How do you think, is it hard to write new Transport that consumes from
> multiple brokers?
>
> S.
>
>
> ttmdev wrote:
>>
>> Yes, I got similar results in my consumer testing. What version of AMQ
>> are you using?
>>
>> Your only recourse may be to multi thread your consumer and have it
>> create a connection to each of the brokers.
>>
>
>
--
View this message in context: http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836p21729358.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Single consumer for multiple brokers?
Posted by Stepan Koltsov <yo...@mx1.ru>.
I'm playing with 5.0.
How do you think, is it hard to write new Transport that consumes from
multiple brokers?
S.
ttmdev wrote:
>
> Yes, I got similar results in my consumer testing. What version of AMQ are
> you using?
>
> Your only recourse may be to multi thread your consumer and have it create
> a connection to each of the brokers.
>
--
View this message in context: http://www.nabble.com/Single-consumer-for-multiple-brokers--tp15768836s2354p15769009.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.