[jira] [Closed] (AMQ-4208) Stuck Message Issue with ActiveMQ 5.6

["Timothy Bish (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Timothy Bish closed AMQ-4208.
-----------------------------
    Resolution: Incomplete

No test case to validate against a current broker.

> Stuck Message Issue with ActiveMQ 5.6
> -------------------------------------
>
>                 Key: AMQ-4208
>                 URL: https://issues.apache.org/jira/browse/AMQ-4208
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-pool, Broker
>    Affects Versions: 5.6.0
>         Environment: Production
>            Reporter: Sinu Sekhar
>            Priority: Critical
>         Attachments: activemq-orig.11302012.xml, activemqthreads6.txt, threads6.txt
>
>
> We have been facing a stuck message issue with 5.6 for a while now. This week I got a chance to reproduce this in a non-production environment every time I ran a high load.
> We use Spring DMLC with the following configuration.
>     <bean id="asyncServiceContainer2" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
>         <property name="connectionFactory" ref="asyncServiceConnectionFactory2" />
>         <property name="destination" ref="asyncServiceDestination" />
>         <property name="messageListener" ref="asyncService" />
>         <!-- Cache JMS connection -->
>         <property name="cacheLevelName" value="CACHE_CONNECTION" />
>         <!-- The maximum number of concurrent consumers to create -->
>         <property name="maxConcurrentConsumers" value="30" />
>         <property name="receiveTimeout" value="10000" />
>         <property name="maxMessagesPerTask" value="20" />
>         <property name="idleTaskExecutionLimit" value="5" />
>     </bean>
> The broker uses the default settings with producerFlowControl on.
> If I send a jmeter load with 100 concurrent users, with 1000 requests from each user...it generates a throughput of around 60TPS/node. This is for the producer side. 
> I see no issues from a producer standpoint. 
> But everytime I run this test, it reaches a point where the deque count just freezes up and the enqueue keeps incrementing because the producer is not affected. This freeze affects all consumers. The only way to unlock it is by restarting the broker. I have seen it being released by restarting the app as well.
> When this free happens, there is usually a large number of *.log files in the kahadb folder and the size of the b-tree file db.data also keeps incrementing indicating the backlog. 
> I have taken jstack traces for the broker as well as the application, which I will attach with this request. 
> Please note that this happens everytime I run with the setup shown.
> One way I can prevent the issue from happening so frequently, is by turning off dynamic scaling by using concurrentConsumers to a high number and setting maxMessagesPerTask to the default (-1 or infinity) for Spring DMLC. This keeps the number of consumers constant and increased the consumption rate, preventing the backlog from happening. In addition to these changes I also had to switch from CACHE_CONNECTION cache type to CACHE_CONSUMER to prevent this from happening.
> I have seen a similar issue reported for an earlier version at http://mail-archives.apache.org/mod_mbox/activemq-users/201005.mbox/%3Cg2v4e6f67721005071419u5ca08751h569775ee4e682c5e@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)