Troubleshooting network of brokers?

[Keith Thornhill <ke...@raptr.com>]

Hi all,

is there any documentation for best practices when setting up a
network of brokers?  this page
(http://activemq.apache.org/networks-of-brokers.html) doesn't really
tell you anything useful.  using 5.3.1 i'm seeing lots of messages
getting stuck on brokers with no relevant consumers, even tho my TTL
is high, as well as brokers passing along a message even tho it has a
local consumer.

i am unsure about the expected behavior for NoB (what works great,
where are there bugs, etc) and would love some feedback from users who
successfully run distributed queues.

if you'd like to read specifics, details are below.

thanks!
-keith

details:
------
my goal is to run distributed worker queues.  for example, a message
could be put on any broker, and it will round-robin it amongst the
current set of consumers.  so ideally, all brokers in a given data
center are connected to all other brokers to allow multiple routes to
consumers.  cross-datacenter connections would be more hub/spoke style
to reduce the number of cross-colo connections.  so imagine 2 clusters
of brokers, with a single connection between the clusters.

i've seen that for a network such as A <-> B <-> C with consumers on B
and C, if i put 10,000 messages on A, they all get sent to B, about
1000 get sent to the local consumer and the other 9000 get sent to C.
since messages can't go backwards, they are stuck there until the
local consumer gets to them.  i'd like it so that each consumer gets
the same number of messages, and thought that the round-robin behavior
is default.

i've also seen that for more complex topologies with more than one
route to a consumer, messages will get stuck on a broker even though
they have no consumers there. dynamicOnly is set to true, and my TTL
is set to 2x the number of brokers.  i also run topics on this network
of brokers, so turning off conduitSubscriptions doesn't seem like an
option. are there known bugs with regards to certain topology
patterns?

Re: Troubleshooting network of brokers?

[Roland Thomas Lichti <ro...@telefonica.de>]

Hi Felix,

fehm schrieb:
> From the 2.paragraph of your details I would say that if you want your all
> consumers to receive all the same sent messages then you should use topics
> instead of queues.
>   
No, he wants every consumer to receive the same workload. Not the same
messages.

bye,
    Roland

-- 
Roland Thomas Lichti
IT Application Manager

Telefónica o2 Germany GmbH & Co. OHG
Hülshorstweg 30, 33415 Verl
t: +49 5246 80 1121, f: +49 5246 80 2121
m: +49 160 98949570
roland.lichti@telefonica.de
http://www.telefonica.de

Bitte finden Sie hier die
handelsrechtlichen Pflichtangaben: http://www.telefonica.de/pflichtangaben.html

Re: Troubleshooting network of brokers?

[fehm <Fe...@cern.ch>]

Hi,

>From the 2.paragraph of your details I would say that if you want your all
consumers to receive all the same sent messages then you should use topics
instead of queues.


Cheers,
Felix



Keith Thornhill-3 wrote:
> 
> Hi all,
> 
> is there any documentation for best practices when setting up a
> network of brokers?  this page
> (http://activemq.apache.org/networks-of-brokers.html) doesn't really
> tell you anything useful.  using 5.3.1 i'm seeing lots of messages
> getting stuck on brokers with no relevant consumers, even tho my TTL
> is high, as well as brokers passing along a message even tho it has a
> local consumer.
> 
> i am unsure about the expected behavior for NoB (what works great,
> where are there bugs, etc) and would love some feedback from users who
> successfully run distributed queues.
> 
> if you'd like to read specifics, details are below.
> 
> thanks!
> -keith
> 
> details:
> ------
> my goal is to run distributed worker queues.  for example, a message
> could be put on any broker, and it will round-robin it amongst the
> current set of consumers.  so ideally, all brokers in a given data
> center are connected to all other brokers to allow multiple routes to
> consumers.  cross-datacenter connections would be more hub/spoke style
> to reduce the number of cross-colo connections.  so imagine 2 clusters
> of brokers, with a single connection between the clusters.
> 
> i've seen that for a network such as A <-> B <-> C with consumers on B
> and C, if i put 10,000 messages on A, they all get sent to B, about
> 1000 get sent to the local consumer and the other 9000 get sent to C.
> since messages can't go backwards, they are stuck there until the
> local consumer gets to them.  i'd like it so that each consumer gets
> the same number of messages, and thought that the round-robin behavior
> is default.
> 
> i've also seen that for more complex topologies with more than one
> route to a consumer, messages will get stuck on a broker even though
> they have no consumers there. dynamicOnly is set to true, and my TTL
> is set to 2x the number of brokers.  i also run topics on this network
> of brokers, so turning off conduitSubscriptions doesn't seem like an
> option. are there known bugs with regards to certain topology
> patterns?
> 
> 


-----
Felix Ehm,
www.cern.ch
-- 
View this message in context: http://old.nabble.com/Troubleshooting-network-of-brokers--tp28396256p28431966.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Troubleshooting network of brokers?

[ttmgary <ga...@ttmsolutions.com>]

Keith,

Regarding your A <-> B <-> C config, I also get a roughly 10/90 distribution
when testing 5.3.1. On 5.3.0, I have randomly seen both 50/50 and 10/90, so
I am not sure what is going on there.

In your "more complex topologies" scenario, I bet you are running into 
https://issues.apache.org/activemq/browse/AMQ-2135 AMQ-2135 . Look at the
last update of that JIRA where ying suggests
suppressDuplicateQueueSubscriptions="true" -- that setting can really make a
difference for stuck messages!

Gary


Keith Thornhill-3 wrote:
> 
> 
> i've seen that for a network such as A <-> B <-> C with consumers on B
> and C, if i put 10,000 messages on A, they all get sent to B, about
> 1000 get sent to the local consumer and the other 9000 get sent to C.
> since messages can't go backwards, they are stuck there until the
> local consumer gets to them.  i'd like it so that each consumer gets
> the same number of messages, and thought that the round-robin behavior
> is default.
> 
> i've also seen that for more complex topologies with more than one
> route to a consumer, messages will get stuck on a broker even though
> they have no consumers there. dynamicOnly is set to true, and my TTL
> is set to 2x the number of brokers.  i also run topics on this network
> of brokers, so turning off conduitSubscriptions doesn't seem like an
> option. are there known bugs with regards to certain topology
> patterns?
> 
> 


-----
ActiveMQ Reference Guide:  http://bit.ly/AMQRefGuide
http://bit.ly/AMQRefGuide 
-- 
View this message in context: http://old.nabble.com/Troubleshooting-network-of-brokers--tp28396256p28467892.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.