[jira] [Commented] (AMQ-4208) Stuck Message Issue with ActiveMQ 5.6

["Sinu Sekhar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510934#comment-13510934 ] 

Sinu Sekhar commented on AMQ-4208:
----------------------------------

Just want to add that I used two machines to reproduce this on a non-prod environment, one for the broker and one for the app. JMeter was run on a different machine, to send load to the application which acted like a producer as well as consumer ... all part of the same war file, implemented using Spring.

                
> Stuck Message Issue with ActiveMQ 5.6
> -------------------------------------
>
>                 Key: AMQ-4208
>                 URL: https://issues.apache.org/jira/browse/AMQ-4208
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-pool, Broker
>    Affects Versions: 5.6.0
>         Environment: Production
>            Reporter: Sinu Sekhar
>            Priority: Critical
>         Attachments: activemq-orig.11302012.xml, activemqthreads6.txt, threads6.txt
>
>
> We have been facing a stuck message issue with 5.6 for a while now. This week I got a chance to reproduce this in a non-production environment every time I ran a high load.
> We use Spring DMLC with the following configuration.
>     <bean id="asyncServiceContainer2" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
>         <property name="connectionFactory" ref="asyncServiceConnectionFactory2" />
>         <property name="destination" ref="asyncServiceDestination" />
>         <property name="messageListener" ref="asyncService" />
>         <!-- Cache JMS connection -->
>         <property name="cacheLevelName" value="CACHE_CONNECTION" />
>         <!-- The maximum number of concurrent consumers to create -->
>         <property name="maxConcurrentConsumers" value="30" />
>         <property name="receiveTimeout" value="10000" />
>         <property name="maxMessagesPerTask" value="20" />
>         <property name="idleTaskExecutionLimit" value="5" />
>     </bean>
> The broker uses the default settings with producerFlowControl on.
> If I send a jmeter load with 100 concurrent users, with 1000 requests from each user...it generates a throughput of around 60TPS/node. This is for the producer side. 
> I see no issues from a producer standpoint. 
> But everytime I run this test, it reaches a point where the deque count just freezes up and the enqueue keeps incrementing because the producer is not affected. This freeze affects all consumers. The only way to unlock it is by restarting the broker. I have seen it being released by restarting the app as well.
> When this free happens, there is usually a large number of *.log files in the kahadb folder and the size of the b-tree file db.data also keeps incrementing indicating the backlog. 
> I have taken jstack traces for the broker as well as the application, which I will attach with this request. 
> Please note that this happens everytime I run with the setup shown.
> One way I can prevent the issue from happening so frequently, is by turning off dynamic scaling by using concurrentConsumers to a high number and setting maxMessagesPerTask to the default (-1 or infinity) for Spring DMLC. This keeps the number of consumers constant and increased the consumption rate, preventing the backlog from happening. In addition to these changes I also had to switch from CACHE_CONNECTION cache type to CACHE_CONSUMER to prevent this from happening.
> I have seen a similar issue reported for an earlier version at http://mail-archives.apache.org/mod_mbox/activemq-users/201005.mbox/%3Cg2v4e6f67721005071419u5ca08751h569775ee4e682c5e@mail.gmail.com%3E

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira