ActiveMQ 5.2 - broker stops sending messages

[Ron Reynolds <te...@ymail.com>]

i know it's a vague subject line and i'm almost certain it's been asked before but i'm under a tight deadline on a near-production system and i need help.  the situation is this:

- single broker (5.2) with only 4 queues (not topics) 
- default broker config with prefetchPolicy.queuePrefetch=0 added to broker's connection pool brokerURL attribute
- single client using failover URL (in anticipation of a second broker once the system is stable) 
- client using Spring-JMS.  
- client-ack, small messages, 7-second retry on the client, using PooledConnectionFactory wrapped around ActiveMQConnectionFactory  
- no transaction management added.  
- both client and server are Linux (CentOS 5.x i think) w/ Sun JVM 1.5.0_11.

the situation is that the system, after running well for an hour or more (typically not more than 24) it will suddenly stop sending notification to the client of new messages in one of the queues (need for it to repro to test if it happens for other queues).  a QueueBrowser shows the messages just fine, but a receiveMessages() with null selection criteria times out before any messages are returned.  i'm able to insert new messages into the queue but the client still gets no notification.

another developer is currently looking at OpenMQ or other JMS implementations to avoid this issue but it has to be a config issue, yes?  


      

Re: ActiveMQ 5.2 - broker stops sending messages

[Bruce Snyder <br...@gmail.com>]

On Mon, Mar 23, 2009 at 1:53 PM, Ron Reynolds <te...@ymail.com> wrote:
> i know it's a vague subject line and i'm almost certain it's been asked before but i'm under a tight deadline on a near-production system and i need help.  the situation is this:
>
> - single broker (5.2) with only 4 queues (not topics)
> - default broker config with prefetchPolicy.queuePrefetch=0 added to broker's connection pool brokerURL attribute
> - single client using failover URL (in anticipation of a second broker once the system is stable)
> - client using Spring-JMS.
> - client-ack, small messages, 7-second retry on the client, using PooledConnectionFactory wrapped around ActiveMQConnectionFactory
> - no transaction management added.
> - both client and server are Linux (CentOS 5.x i think) w/ Sun JVM 1.5.0_11.
>
> the situation is that the system, after running well for an hour or more (typically not more than 24) it will suddenly stop sending notification to the client of new messages in one of the queues (need for it to repro to test if it happens for other queues).  a QueueBrowser shows the messages just fine, but a receiveMessages() with null selection criteria times out before any messages are returned.  i'm able to insert new messages into the queue but the client still gets no notification.
>
> another developer is currently looking at OpenMQ or other JMS implementations to avoid this issue but it has to be a config issue, yes?

In the conf/activemq.xml file, what is the memoryUsage limit set at
currently? If it's still at the default of 20mb, I suggest raising it
to a more appropriate limit. This is the memory limit for the entire
broker and should be big enough to hold all messages in memory at any
given point in time. Remember that this needs to remain in accordance
with the JVM memory settings ( the java -Xmx setting).

Also, when the broker is not dispatching messages to clients, grab a
thread dump to see where exactly it's hanging. A thread dump can be
triggered using a kill -QUIT <pid> on Unix or a ctrl-break on Windows
or the jstack utility.

There's also an open issue for something similar to this in AMQ-2009:

http://issues.apache.org/activemq/browse/AMQ-2009

Even so, I still highly suggest getting a thread dump to see where
it's hanging.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/

Blog: http://bruceblog.org/

Re: ActiveMQ 5.2 - broker stops sending messages

[Rob Davies <ra...@gmail.com>]

This looks similar to a known issue - AMQ-2009  - can you try the  
latest 5.3 snaphot ?


cheers,

Rob

Rob Davies
http://fusesource.com
http://rajdavies.blogspot.com/





On 23 Mar 2009, at 19:53, Ron Reynolds wrote:

> i know it's a vague subject line and i'm almost certain it's been  
> asked before but i'm under a tight deadline on a near-production  
> system and i need help.  the situation is this:
>
> - single broker (5.2) with only 4 queues (not topics)
> - default broker config with prefetchPolicy.queuePrefetch=0 added to  
> broker's connection pool brokerURL attribute
> - single client using failover URL (in anticipation of a second  
> broker once the system is stable)
> - client using Spring-JMS.
> - client-ack, small messages, 7-second retry on the client, using  
> PooledConnectionFactory wrapped around ActiveMQConnectionFactory
> - no transaction management added.
> - both client and server are Linux (CentOS 5.x i think) w/ Sun JVM  
> 1.5.0_11.
>
> the situation is that the system, after running well for an hour or  
> more (typically not more than 24) it will suddenly stop sending  
> notification to the client of new messages in one of the queues  
> (need for it to repro to test if it happens for other queues).  a  
> QueueBrowser shows the messages just fine, but a receiveMessages()  
> with null selection criteria times out before any messages are  
> returned.  i'm able to insert new messages into the queue but the  
> client still gets no notification.
>
> another developer is currently looking at OpenMQ or other JMS  
> implementations to avoid this issue but it has to be a config issue,  
> yes?
>
>