You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by watcher <pa...@daly.ws> on 2010/12/07 16:46:07 UTC

Message stuck on queue

We are currently experience a problem with activemq version 5.4.0 
Every 2-4 days of testing we get 1 message stuck on the queue. 
We have approx 10 queues were we serialize a POJO onto each queue. 
We have no more than 1000 messages on each queue.

We have up to 5 consumers (slow consumers between 5secs and 1min) on each
queue.

Each client will wait forever on the queue until the queue is empty.
 consumerTemplate.receive(MY_QUEUE);

Our client connect with a prefetch = 1, we also tried value of 0
failover:(tcp://127.0.0.1:61616)?jms.prefetchPolicy.queuePrefetch=1


<code>
   <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="mainBroker" 
	dataDirectory="${activemq.base}/data"
destroyApplicationContextOnStop="true" useJmx="true">
		
        <!--
			For better performances use VM cursor and small memory limit.
			For more information, see:
            
            http://activemq.apache.org/message-cursors.html
            
            Also, if your producer is "hanging", it's probably due to
producer flow control.
            For more information, see:
            http://activemq.apache.org/producer-flow-control.html
        -->
              
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true"
memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true"
memoryLimit="1mb">
                  <!-- Use VM cursor for better latency
                       For more information, see:
                       
                       http://activemq.apache.org/message-cursors.html
                       
                  <pendingQueuePolicy>
                    <vmQueueCursor/>
                  </pendingQueuePolicy>
                  -->
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy> 
 
        
        <!-- 
            The managementContext is used to configure how ActiveMQ is
exposed in 
            JMX. By default, ActiveMQ uses the MBean server that is started
by 
            the JVM. For more information, see: 
            
            http://activemq.apache.org/jmx.html 
        -->
        <managementContext>
            <managementContext connectorPort="2011"
jmxDomainName="my-broker" />
        </managementContext>

        <!-- 
            Configure message persistence for the broker. The default
persistence
            mechanism is the KahaDB store (identified by the kahaDB tag). 
            For more information, see: 
            
            http://activemq.apache.org/persistence.html 
        -->
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
        
          <plugins>
			<statisticsBrokerPlugin/>
		</plugins>
          <!--
            The systemUsage controls the maximum amount of space the broker
will 
            use before slowing down producers. For more information, see:
            
            http://activemq.apache.org/producer-flow-control.html
           -->  
         <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="256 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="300 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
		
		 
        <!-- 
            The transport connectors expose ActiveMQ over a given protocol
to
            clients and other brokers. For more information, see: 
            
            http://activemq.apache.org/configuring-transports.html 
        -->
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>
		

    </broker>

    <!-- 
        Uncomment to enable Camel
        Take a look at activemq-camel.xml for more details
	
  -->        
  	<import resource="camel.xml"/>
  

    <!-- 
        Enable web consoles, REST and Ajax APIs and demos
        Take a look at activemq-jetty.xml for more details 
    -->
    <import resource="jetty.xml"/>
    
</code>

two different setups 
server windows 2008 
client windows 7

and
server: windows server 2003
client : windows 7 

When we restart activemq service the message appears.

Is there anything that we can do to prevent this from happening?
Thanks



-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3076607.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by "lernen.2007" <er...@googlemail.com>.
I check it with the latest activemq snapshot and it is the same attitude -->
the messages stuck in queue. We want to go in production because of this bug
we can't.
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3255085.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by Norman Maurer <no...@apache.org>.
I think it would be better todo something like that for the fix:

    byte[] content;
    if (msg.getText== null) {
        content = new byte[0];
    } else {
        content = msg.getText().getBytes("UTF-8");
    }
    command.setContent(content);

But I'm not to familiar with stomp..

Bye,
Norman


2011/2/1 anton <ju...@gmail.com>:
>
> We are using ActiveMQ 5.4.2 on CentOS 5.4 and also run into a "message stuck
> in a queue" scenario. In our case these stuck messages had no content.
> Here's an outline of what was happening:
>
> a) we were seeing a few empty messages submitted to the queue (our fault -
> there weren't really supposed to be empty, but because of a producer error
> they were)
> b) our consumer application (using STOMP) would never get these empty
> messages and would wait in perpetuity
> c) deleteing these empty messages from the queue and reconnecting the
> consumer would resume the flow of messages
> d) In ActiveMQ log, we noticed this stack trace:
>
> 2011-01-30 17:12:34,811 | ERROR | Failed to page in more queue messages  |
> org.apache.activemq.broker.region.Queue | Queue:test66
> java.lang.NullPointerException
>        at
> org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertMessage(LegacyFrameTranslator.java:90)
>        at
> org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:610)
>        at
> org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:93)
>        at
> org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:592)
>        at
> org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:58)
>        at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>        at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1249)
>        at
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:810)
>        at
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:649)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:599)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:156)
>        at
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1798)
>        at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1745)
>        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>        at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>        at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
>
>
> e) Putting a try/catch block around that line in LegacyFrameTranslator.java
> (activemq-core/src/main/java/org/apache/activemq/transport/stomp/LegacyFrameTranslator.java:90)
> causes the messages to not be stuck any more: they now show up successfully
> on the consumer side. Here's the change we did:
>
>
> 90c90,94
> <             command.setContent(msg.getText().getBytes("UTF-8"));
> ---
>>             try {
>>                 command.setContent(msg.getText().getBytes("UTF-8"));
>>             } catch (Throwable e) {
>>                 command.setContent(new byte[0]);
>>             }
>
>
> Not sure if this is the "right" way to fix the problem.
> Hoping someone on this list could shed some light.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3252992.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Message stuck on queue

Posted by anton <ju...@gmail.com>.
We are using ActiveMQ 5.4.2 on CentOS 5.4 and also run into a "message stuck
in a queue" scenario. In our case these stuck messages had no content.
Here's an outline of what was happening:

a) we were seeing a few empty messages submitted to the queue (our fault -
there weren't really supposed to be empty, but because of a producer error
they were)
b) our consumer application (using STOMP) would never get these empty
messages and would wait in perpetuity
c) deleteing these empty messages from the queue and reconnecting the
consumer would resume the flow of messages
d) In ActiveMQ log, we noticed this stack trace: 

2011-01-30 17:12:34,811 | ERROR | Failed to page in more queue messages  |
org.apache.activemq.broker.region.Queue | Queue:test66
java.lang.NullPointerException
	at
org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertMessage(LegacyFrameTranslator.java:90)
	at
org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:610)
	at
org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:93)
	at
org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:592)
	at
org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:58)
	at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
	at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1249)
	at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:810)
	at
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
	at
org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:649)
	at
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:599)
	at
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:156)
	at
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1798)
	at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1745)
	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
	at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
	at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
	at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)


e) Putting a try/catch block around that line in LegacyFrameTranslator.java
(activemq-core/src/main/java/org/apache/activemq/transport/stomp/LegacyFrameTranslator.java:90)
causes the messages to not be stuck any more: they now show up successfully
on the consumer side. Here's the change we did:


90c90,94
<             command.setContent(msg.getText().getBytes("UTF-8"));
---
>             try {
>                 command.setContent(msg.getText().getBytes("UTF-8"));
>             } catch (Throwable e) {
>                 command.setContent(new byte[0]);
>             }


Not sure if this is the "right" way to fix the problem. 
Hoping someone on this list could shed some light.

-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3252992.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by "lernen.2007" <er...@googlemail.com>.
I will test with the 5.5 snapshot and inform you about that.
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3251328.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by Gary Tully <ga...@gmail.com>.
I think you may be experiencing
https://issues.apache.org/jira/browse/AMQ-3167 - if you get a chance
you can validate a 5.5-SNAPSHOT with the cache enabled?

On 1 February 2011 12:00, watcher <pa...@daly.ws> wrote:
>
> We havnt been able to reproduce the bug after we turned off caching.
>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb"
> useCache="false">
>
> We were using version 5.2.0 in another application and never experience this
> problem before but I think that is because they changed the persistence
> storage to kahadb in 5.3 by default.
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3251012.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
http://blog.garytully.com
http://fusesource.com

Re: Message stuck on queue

Posted by watcher <pa...@daly.ws>.
We havnt been able to reproduce the bug after we turned off caching.

<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb" 
useCache="false"> 

We were using version 5.2.0 in another application and never experience this
problem before but I think that is because they changed the persistence
storage to kahadb in 5.3 by default.
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3251012.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by "lernen.2007" <er...@googlemail.com>.
I have the same problem with message stuck on queue. I use as persistence a
oracle database with jdbc. I think that is a bug in activemq. The version
5.3.1 hasn't any problem.
-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3250847.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message stuck on queue

Posted by Rob Davies <ra...@gmail.com>.
For statistics - can you should be able to use wildcards - if you use '>' as the destination name - what do you get ?

On 10 Dec 2010, at 22:23, watcher wrote:

> 
> An update so far.
> 
> We stilll cant seem to find a resolution to this issue. We have noticed
> similar user having problems.
> See here. 
> http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html
> http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html 
> 
> We implemented a similar workaround. Whenever we get to the end on the queue
> and a message is stuck we purge the queue a republish the message. This is
> not a perfect solution as you can imagine.
> 
> We tried upgrading to 5.4.1 but the error still exists. Since then 5.4.2 has
> been released but we're not convinced that this will solve the issue but we
> will try soon.
> 
> Just another bit of info, we use the statisics plugin via camel to determine
> the queue size however I've not been able to implement a wildcard scan of
> queues so we ended up hardcoding a route for each queue that may or may not
> exist up to a max of 60 queues. The only errors we see in activemq logs are
> related to timeout when trying to get queue stats for queues that do not
> exist.
> 
> http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 
> http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 
> 
> Is there any logging we could enable to understand why a message will not
> get processed?
> 
> We getting to the stage of replacing activemq with another messaging
> provider but I like to think this could be solved. Any advise would be very
> helpful.
> 
> Thanks.
> 
> -- 
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3082780.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: Message stuck on queue

Posted by watcher <pa...@daly.ws>.
An update so far.

We stilll cant seem to find a resolution to this issue. We have noticed
similar user having problems.
See here. 
http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html
http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html 

We implemented a similar workaround. Whenever we get to the end on the queue
and a message is stuck we purge the queue a republish the message. This is
not a perfect solution as you can imagine.

We tried upgrading to 5.4.1 but the error still exists. Since then 5.4.2 has
been released but we're not convinced that this will solve the issue but we
will try soon.

Just another bit of info, we use the statisics plugin via camel to determine
the queue size however I've not been able to implement a wildcard scan of
queues so we ended up hardcoding a route for each queue that may or may not
exist up to a max of 60 queues. The only errors we see in activemq logs are
related to timeout when trying to get queue stats for queues that do not
exist.

http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 
http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 

Is there any logging we could enable to understand why a message will not
get processed?

We getting to the stage of replacing activemq with another messaging
provider but I like to think this could be solved. Any advise would be very
helpful.

Thanks.

-- 
View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3082780.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.