Trace Level Logging For Lost Messages

[greenbean <Ke...@ngc.com>]

We are using Activemq 4.1.1 with Stomp.  We can reproduce a condition where
it appears messages are lost.  However, we would like to trace the internal
flow of messages to see exactly what is happening.  With a standalone
Activemq instance running, adjusting log4j.properties to debug as suggested
online and in the properties file produces very little logging about what is
happening.  Is there a better way to get very detailed logging information?

We appear to lose messages when we have around 100 Stomp consumers
distributed to several machines consuming from a single queue using
different message selectors.  The remote consumers send a message to another
queue after processing the data in the message they consumed.  There are
multiple processes producing messages that are consumed by the remote
consumers.

Things appear to work fine until a simulated level of parallelism reaches a
certain point.  This means each producer process produces, say, 24 messages
with different header properties for consumption by remote consumers with
specific message selectors instead of 10 messages with different header
properties.  A point is reached after a couple of cycles of production and
consumption where activemq stops functioning.  Messages are no longer
delivered after this point.  Using a test driver to send a message to a
queue appears to send a message.  However, the message is not delivered, and
the queue size shown through JMX does not increase.  Once Activemq enters
this state, the only way to correct the issue is to stop activemq, and
restart it.  However, the problem appears again after a couple more cycles
of production and consumption of messages.

Any suggestions or help on tracking this issue would be great.
-- 
View this message in context: http://www.nabble.com/Trace-Level-Logging-For-Lost-Messages-tf3607583s2354.html#a10079347
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Trace Level Logging For Lost Messages

[James Strachan <ja...@gmail.com>]

On 4/19/07, greenbean <Ke...@ngc.com> wrote:
>
> We are using Activemq 4.1.1 with Stomp.  We can reproduce a condition where
> it appears messages are lost.  However, we would like to trace the internal
> flow of messages to see exactly what is happening.  With a standalone
> Activemq instance running, adjusting log4j.properties to debug as suggested
> online and in the properties file produces very little logging about what is
> happening.  Is there a better way to get very detailed logging information?

We welcome patches if you want to try alter the logging behaviour to
expose more information you need.


> We appear to lose messages when we have around 100 Stomp consumers
> distributed to several machines consuming from a single queue using
> different message selectors.  The remote consumers send a message to another
> queue after processing the data in the message they consumed.  There are
> multiple processes producing messages that are consumed by the remote
> consumers.
>
> Things appear to work fine until a simulated level of parallelism reaches a
> certain point.  This means each producer process produces, say, 24 messages
> with different header properties for consumption by remote consumers with
> specific message selectors instead of 10 messages with different header
> properties.  A point is reached after a couple of cycles of production and
> consumption where activemq stops functioning.  Messages are no longer
> delivered after this point.  Using a test driver to send a message to a
> queue appears to send a message.  However, the message is not delivered, and
> the queue size shown through JMX does not increase.  Once Activemq enters
> this state, the only way to correct the issue is to stop activemq, and
> restart it.  However, the problem appears again after a couple more cycles
> of production and consumption of messages.
>
> Any suggestions or help on tracking this issue would be great.

When things seem to hang, I wonder if there's some kinda deadlock
found? e.g. could you try create a thread dump to see if there's a
deadlock?

Also its sometimes worth experimenting with StompConnect with
ActiveMQ; which is an alternative Stomp transport using the JMS API
underneath to see if it has the same issues. As this could help to
track down a specific issue with the native Stomp transport for
ActiveMQ.

http://stomp.codehaus.org/StompConnect

-- 
James
-------
http://macstrac.blogspot.com/