Network with slow link

[Anders Bengtsson <nd...@yahoo.se>]

Hi,

I'm trying to create a network of brokers mostly for distributing messages
from one producer to lots of consumers. The producer and half of the
consumers are on one network (A) and the rest of the consumers are on
another network (B). Between A and B is a slow connection.

We want to minimize the number of messages over the slow link, so we don't
want any consumers in B to connect directly to a broker in A. I figured a
network of brokers could solve this.

The brokers will create points of failure on the way to the consumers in B,
so I'll need redundant brokers. How would I configure multiple redundant
brokers in both networks, but still minimizing the number of messages
between A and B?

thanks

/Anders

PS. Please don't refer me to the "network of brokers" documentation page.
I've read it and it isn't much more than an overview.

-- 
View this message in context: http://www.nabble.com/Network-with-slow-link-tf2072072.html#a5704490
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Network with slow link

[Anders Bengtsson <nd...@yahoo.se>]

John Heitmann wrote:
> 
> True loops seemed to be detected ok, but some common non-loop multiple
> paths  
> were not detected. For example if you have a producer to broker A, A  
> connected to brokers B1 and B2, B1 and B2 connected only to broker C,  
> and you consume from C, then your consumer would see 2 copies of each  
> message on a topic.
> 

Yes, the documentation warns that you can get duplicates if you have two
paths to the receiver. I guess it wouldn't be too hard to filter out
duplicates if we'd be forced to. But we experienced true loops, both with
our own and AMQ's internal messages.
I'll see if I have the time to try to reproduce it. I'm more worried about
the brokers not working properly after a restart (AMQ-870) right now.

/Anders

-- 
View this message in context: http://www.nabble.com/Network-with-slow-link-tf2072072.html#a5721271
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Network with slow link

[John Heitmann <jh...@gmail.com>]

On Aug 8, 2006, at 6:18 AM, Anders Bengtsson wrote:

> On a related issue, should AMQ detect message loops? Some  
> documentation says
> that it does, by keeping a list of all visited brokers in each  
> message,
> while other documentation (for network TTL) warns about the risk of  
> loops
> with a high network TTL.
> We experienced a few message loops when trying out some more  
> complex broker
> network topologies.

The last I looked into this (late winter, around 4.0-M3 I think)  
there were known issues with some network topologies. True loops  
seemed to be detected ok, but some common non-loop multiple paths  
were not detected. For example if you have a producer to broker A, A  
connected to brokers B1 and B2, B1 and B2 connected only to broker C,  
and you consume from C, then your consumer would see 2 copies of each  
message on a topic.

Since we don't use master/slave and we needed multiple brokers on  
each hop for redundancy we use queues on the paths that would give us  
troubles. Queues ensure that only one path gets a message. The new  
virtual destination functionality will make this kind of setup even  
easier.

John

Re: Network with slow link

[James Strachan <ja...@gmail.com>]

On 8/8/06, Anders Bengtsson <nd...@yahoo.se> wrote:
>
>
>
> James.Strachan wrote:
> >
> >
> >> The brokers will create points of failure on the way to the consumers in
> >> B,
> >> so I'll need redundant brokers. How would I configure multiple redundant
> >> brokers in both networks, but still minimizing the number of messages
> >> between A and B?
> >
> > Just run 2 brokers on each side of the slow link so that things will
> > failover if one of the dies?
> >
>
> Yes, that's what we tried. We ran into issues with the brokers not
> re-starting as they should, which I've now reported as AMQ-870.
>
> On a related issue, should AMQ detect message loops? Some documentation says
> that it does, by keeping a list of all visited brokers in each message,

Yes it does.

> while other documentation (for network TTL) warns about the risk of loops
> with a high network TTL.
> We experienced a few message loops when trying out some more complex broker
> network topologies.

If you can create a test case that can reproduce a loop it'd be most helpful.

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Network with slow link

[Anders Bengtsson <nd...@yahoo.se>]

James.Strachan wrote:
> 
> 
>> The brokers will create points of failure on the way to the consumers in
>> B,
>> so I'll need redundant brokers. How would I configure multiple redundant
>> brokers in both networks, but still minimizing the number of messages
>> between A and B?
> 
> Just run 2 brokers on each side of the slow link so that things will
> failover if one of the dies?
> 

Yes, that's what we tried. We ran into issues with the brokers not
re-starting as they should, which I've now reported as AMQ-870.

On a related issue, should AMQ detect message loops? Some documentation says
that it does, by keeping a list of all visited brokers in each message,
while other documentation (for network TTL) warns about the risk of loops
with a high network TTL.
We experienced a few message loops when trying out some more complex broker
network topologies.

/Anders

-- 
View this message in context: http://www.nabble.com/Network-with-slow-link-tf2072072.html#a5706215
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Network with slow link

[James Strachan <ja...@gmail.com>]

On 8/8/06, Anders Bengtsson <nd...@yahoo.se> wrote:
> Hi,
>
> I'm trying to create a network of brokers mostly for distributing messages
> from one producer to lots of consumers. The producer and half of the
> consumers are on one network (A) and the rest of the consumers are on
> another network (B). Between A and B is a slow connection.
>
> We want to minimize the number of messages over the slow link, so we don't
> want any consumers in B to connect directly to a broker in A. I figured a
> network of brokers could solve this.

Yes


> The brokers will create points of failure on the way to the consumers in B,
> so I'll need redundant brokers. How would I configure multiple redundant
> brokers in both networks, but still minimizing the number of messages
> between A and B?

Just run 2 brokers on each side of the slow link so that things will
failover if one of the dies?

-- 

James
-------
http://radio.weblogs.com/0112098/