tracing stuck messages in network of brokers

[aburgel <al...@dailycandy.com>]

hi,

i have a network of 4 brokers. i've created a queue where each broker has 1
producer, and only 1 of the brokers has a consumer. i'm running activemq
5.1.

about every third or fourth message i produce on one node will not end up at
the consumer.

via jmx, i can see that the brokers all have a connection to each other, so
there's no connectivity issue.

i have turned on the logging plugin, so i can see that the message is
created and sent out. when the message is successfully delivered, i see it
go directly to the consumer node. when it gets stuck, it goes to each of the
other two nodes and then stays put at the last one.

if i restart the node where that message is stuck, then it will be delivered
to the consumer. but otherwise it just sits there.

i have a bunch of logs that i can post if its helpful. i modified the
logging broker plugin to log more info, specifically when consumers are
added and removed.

when the nodes start up, i can see that the consumer for this particular
queue is added on all the nodes, tho sometimes the brokerPath is different.

is it possible that the routes between the brokers are looped or in some
other way causing a message to get stuck?

is there a way to get that stuck message to move again? it seems a bit
strange that it would just sit there for ever on a node with no consumer.

i have logs and my config. if that would help, let me know and i'll post it.
any help is much appreciated.

--alex
-- 
View this message in context: http://www.nabble.com/tracing-stuck-messages-in-network-of-brokers-tp17395138s2354p17395138.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: tracing stuck messages in network of brokers

[aburgel <al...@dailycandy.com>]

Apologies for not posting my settings.

I have networkTTL set to 10 (there's only 4 nodes, so this should be more
than enough, and anyways i only see the message go through 2 nodes before it
gets stuck.)

dynamicOnly is true. and duplex is false. since dynamicOnly is true, the
message should only get sent to the broker with the consumer on it. I'm
pretty sure the producer node knows about the consumer because when I start
up I see a ConsumerInfo message arriving with a brokerPath that starts with
the node that contains the consumer. I actually get a few of these
ConsumerInfo messages. Some having longer paths than others, but they all
begin with the correct node.

the bug you post below looks like it could be relevent even tho its for
duplex connections. i will try a snapshot with that fix.

thanks.


ttmdev wrote:
> 
> What do you have networkTTL set at?
> 
> Also, depending on how the brokers are interconnected, you may want to
> consider setting dynamicOnly to true; this ensures that messages are only
> routed to brokers with active consumers. 
> 
> Careful using 'duplex' bridges between brokers in combination with high
> networkTTL settings. Messages will bounce around needlessly. This has been
> patched, but make sure you have it. 
> 
> http://issues.apache.org/activemq/browse/AMQ-1661
> 
> Hope this helps,  
> Joe
> Get a free ActiveMQ user guide at www.ttmsolutions.com 
> 
> 
> aburgel wrote:
>> 
>> hi,
>> 
>> i have a network of 4 brokers. i've created a queue where each broker has
>> 1 producer, and only 1 of the brokers has a consumer. i'm running
>> activemq 5.1.
>> 
>> about every third or fourth message i produce on one node will not end up
>> at the consumer.
>> 
>> via jmx, i can see that the brokers all have a connection to each other,
>> so there's no connectivity issue.
>> 
>> i have turned on the logging plugin, so i can see that the message is
>> created and sent out. when the message is successfully delivered, i see
>> it go directly to the consumer node. when it gets stuck, it goes to each
>> of the other two nodes and then stays put at the last one.
>> 
>> if i restart the node where that message is stuck, then it will be
>> delivered to the consumer. but otherwise it just sits there.
>> 
>> i have a bunch of logs that i can post if its helpful. i modified the
>> logging broker plugin to log more info, specifically when consumers are
>> added and removed.
>> 
>> when the nodes start up, i can see that the consumer for this particular
>> queue is added on all the nodes, tho sometimes the brokerPath is
>> different.
>> 
>> is it possible that the routes between the brokers are looped or in some
>> other way causing a message to get stuck?
>> 
>> is there a way to get that stuck message to move again? it seems a bit
>> strange that it would just sit there for ever on a node with no consumer.
>> 
>> i have logs and my config. if that would help, let me know and i'll post
>> it. any help is much appreciated.
>> 
>> --alex
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/tracing-stuck-messages-in-network-of-brokers-tp17395138s2354p17396807.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: tracing stuck messages in network of brokers

[ttmdev <jo...@ttmsolutions.com>]

What do you have networkTTL set at?

Also, depending on how the brokers are interconnected, you may want to
consider setting dynamicOnly to true; this ensures that messages are only
routed to brokers with active consumers. 

Careful using 'duplex' bridges between brokers in combination with high
networkTTL settings. Messages will bounce around needlessly. This has been
patched, but make sure you have it. 

http://issues.apache.org/activemq/browse/AMQ-1661

Hope this helps,  
Joe
Get a free ActiveMQ user guide at www.ttmsolutions.com 


aburgel wrote:
> 
> hi,
> 
> i have a network of 4 brokers. i've created a queue where each broker has
> 1 producer, and only 1 of the brokers has a consumer. i'm running activemq
> 5.1.
> 
> about every third or fourth message i produce on one node will not end up
> at the consumer.
> 
> via jmx, i can see that the brokers all have a connection to each other,
> so there's no connectivity issue.
> 
> i have turned on the logging plugin, so i can see that the message is
> created and sent out. when the message is successfully delivered, i see it
> go directly to the consumer node. when it gets stuck, it goes to each of
> the other two nodes and then stays put at the last one.
> 
> if i restart the node where that message is stuck, then it will be
> delivered to the consumer. but otherwise it just sits there.
> 
> i have a bunch of logs that i can post if its helpful. i modified the
> logging broker plugin to log more info, specifically when consumers are
> added and removed.
> 
> when the nodes start up, i can see that the consumer for this particular
> queue is added on all the nodes, tho sometimes the brokerPath is
> different.
> 
> is it possible that the routes between the brokers are looped or in some
> other way causing a message to get stuck?
> 
> is there a way to get that stuck message to move again? it seems a bit
> strange that it would just sit there for ever on a node with no consumer.
> 
> i have logs and my config. if that would help, let me know and i'll post
> it. any help is much appreciated.
> 
> --alex
> 

-- 
View this message in context: http://www.nabble.com/tracing-stuck-messages-in-network-of-brokers-tp17395138s2354p17396466.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.