Active MQ, Only one of many MessageListeners stuck while reading still causes queue to grow

[mathias <ma...@afjochnick.net>]

ActiveMQ version 5.10.2

I have a weird behavior on an ActiveMQ queue i can't understand. Not
super-good at ActiveMQ so i'm hoping that someone can help out.

I have a queue, configured in tomcat. I have a MessageListener configured
through Spring with 3 instances. They basically do an HTTP request, process
the answer and put the answer on another queue.

Yesterday, i had connectivity issues, so the (SOAP) requests timed out. When
that happens, I throw a custom exception, catch it in "onMessage" and send
an error message on the response queue. I can then see that the listener
took another message off the queue, and then hung. No exception or anything.
Why this happened i'm not sure of, i'm investigating that.

But here's the strange thing:

This happened to 2 of my three MessageListeners. They stopped, but my third
one still reads from the queue. HOWEVER, the queue is growing and the
queuesize was more than 500 messages before i restarted the server.

The two listeners that were hung each had a "dispatchedqueuesize" of about
half of that each.

So, to me it seems that the messages that were put on the queue become
"reserved" for each of the listeners, even if they don't actively read from
the queue. I can see that the "active" listener still process messages, so
why doesn't it process every message?? I want the listeners to just read
messages in the order they come in. If only one "is left" it should still
read every message put on the queue. Why isn't it doing that??

Pointers much appreciated.

Spring config for receiving end below:

<amq:connectionFactory id="refConnectionFactory"
brokerURL="${jms.broker.url}" clientID="theclient"/>
<bean id="connectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
    <constructor-arg ref="refConnectionFactory"/>
    <property name="sessionCacheSize" value="10"/>
</bean>


<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
    <constructor-arg ref="connectionFactory"/>
</bean>


<jms:listener-container concurrency="3">
    <jms:listener id="locationListener" destination="location.out"
ref="locationQueryListener"/>
</jms:listener-container>



--
View this message in context: http://activemq.2283324.n4.nabble.com/Active-MQ-Only-one-of-many-MessageListeners-stuck-while-reading-still-causes-queue-to-grow-tp4708872.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Active MQ, Only one of many MessageListeners stuck while reading still causes queue to grow

[Christopher Shannon <ch...@gmail.com>]

What you are seeing is the prefetch setting.  For performance reasons
messages will be dispatched to consumers so that they are waiting on the
client as soon as the client finishes a message.  So those messages that
were sent to the hung consumer will only be released once that consumer
closes.  If you lowered the prefetch value to 1, for example, then messages
would only be sent 1 at a time and the behavior would be what you want but
the trade off would be performance.

You can read more about it here:
http://activemq.apache.org/what-is-the-prefetch-limit-for.html


On Fri, Mar 4, 2016 at 7:22 AM, mathias <ma...@afjochnick.net> wrote:

> ActiveMQ version 5.10.2
>
> I have a weird behavior on an ActiveMQ queue i can't understand. Not
> super-good at ActiveMQ so i'm hoping that someone can help out.
>
> I have a queue, configured in tomcat. I have a MessageListener configured
> through Spring with 3 instances. They basically do an HTTP request, process
> the answer and put the answer on another queue.
>
> Yesterday, i had connectivity issues, so the (SOAP) requests timed out.
> When
> that happens, I throw a custom exception, catch it in "onMessage" and send
> an error message on the response queue. I can then see that the listener
> took another message off the queue, and then hung. No exception or
> anything.
> Why this happened i'm not sure of, i'm investigating that.
>
> But here's the strange thing:
>
> This happened to 2 of my three MessageListeners. They stopped, but my third
> one still reads from the queue. HOWEVER, the queue is growing and the
> queuesize was more than 500 messages before i restarted the server.
>
> The two listeners that were hung each had a "dispatchedqueuesize" of about
> half of that each.
>
> So, to me it seems that the messages that were put on the queue become
> "reserved" for each of the listeners, even if they don't actively read from
> the queue. I can see that the "active" listener still process messages, so
> why doesn't it process every message?? I want the listeners to just read
> messages in the order they come in. If only one "is left" it should still
> read every message put on the queue. Why isn't it doing that??
>
> Pointers much appreciated.
>
> Spring config for receiving end below:
>
> <amq:connectionFactory id="refConnectionFactory"
> brokerURL="${jms.broker.url}" clientID="theclient"/>
> <bean id="connectionFactory"
> class="org.springframework.jms.connection.CachingConnectionFactory">
>     <constructor-arg ref="refConnectionFactory"/>
>     <property name="sessionCacheSize" value="10"/>
> </bean>
>
>
> <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
>     <constructor-arg ref="connectionFactory"/>
> </bean>
>
>
> <jms:listener-container concurrency="3">
>     <jms:listener id="locationListener" destination="location.out"
> ref="locationQueryListener"/>
> </jms:listener-container>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Active-MQ-Only-one-of-many-MessageListeners-stuck-while-reading-still-causes-queue-to-grow-tp4708872.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>