[jira] [Commented] (AMQ-6568) Consumers stop consuming

["Andreas Gies (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMQ-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885623#comment-15885623 ] 

Andreas Gies commented on AMQ-6568:
-----------------------------------

I have provided the heapdumps of two affected containers here:

https://www.dropbox.com/sh/p7nnaz5qyqhabkn/AADFA2AqtBhsiQPhBuLzxDtSa?dl=0

The size of the heap dumps was just too big to attach to JIRA. I hope that way of sharing is ok. 


> Consumers stop consuming 
> -------------------------
>
>                 Key: AMQ-6568
>                 URL: https://issues.apache.org/jira/browse/AMQ-6568
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker, JMS client
>    Affects Versions: 5.14.2
>         Environment: CentOS 6.5, Java 7, 64-bit
>            Reporter: Andreas Gies
>         Attachments: broker.xml, ThreadDump.txt, UnDamaged_ThreadDump.txt.zip
>
>
> We are working on a fairly large distributed project using ActiveMQ as our JMS backend. 
> Currently we are using Version 5.14.2. 
> The application consists of many containers communicating over AMQ.
> All JMS communication endpoints are implemented with Camel 2.17.3. 
> The container itself is an OSGi container on top of Apache Felix, all implemented in Scala.
> The production version is based on Scala 2.10 / JDK 7. The next version will be Scala 2.11 / JDK 8. 
> We are not using Karaf, but have our own bundles to implement container configuration 
> and distribution.
> The overall install base is roughly 2500 containers in as many local site connected to the 
> central data center. The data center does not run AMQ, but a different JMS provider. All local 
> containers use AMQ und bridge into the data center. 
> Our problem is that we observe in ~1% of our containers that messages are not being delivered 
> from AMQ. The Queue Size is > 0 and the consuming Camel Route is started, but no message 
> is dispatched. Message producers can still produce messages, but none are dispatched. 
> In such a scenario no messages are dispatched to any consumers. Restating the route bundles 
> does not have an effect, only restarting the containers rectifies the problem. 
> Our investigation so far has not given any hints what might be special regarding the circumstances 
> when we encounter the problem. We have taken thread dumps from containers in that state, but 
> have not been able to spot the problem. 
> The initial question is whether someone in the community could provide us a hint how we could 
> narrow down the problem. 
> For today I am attaching our AMQ configuration file and also one of latest thread dumps. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)