[jira] Reopened: (AMQ-2009) Problem with message dispatch after a while

["ying (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/activemq/browse/AMQ-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ying reopened AMQ-2009:
-----------------------


I see this in Trunk 771718. Our network of 4 brokers are running for almost 30 days, then we see a broker is not dispatching msgs. It has about 400mb data in its data directory. 

We restart the application to talk to another broker, restart the problematic broker, it starts fine, bridged ok, sees the consumer on the other broker, but all the msgs stuck on this broker is not dispatched. They are stuck.

I will try to gather more information when I get more idea. A simple unit test of the case is ok with less msgs. I suspect this happens when a lot of msgs are stuck on the queue, then it fails to dispatch even after restart. BTW, we have enough memory on the box for the broker to run and jconsole show it is using only a portion. There is no error in the log with debug turned on.

DemandForwardingBridgeSupport's serviceLocalCommand is supposed to be called but not. Any possible threading issue? Regarding the demandforwarding bridge, do you know the name of the thread I shall look in jconsole. due to the complexity of our system, there are about 170 live threads in the jconsole for this broker. Maybe a thread is blocked. 

Any suggestion is welcome. I am looking into this issue until i find a fix. 

> Problem with message dispatch after a while
> -------------------------------------------
>
>                 Key: AMQ-2009
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2009
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0, 5.2.0
>            Reporter: Rajani Chennamaneni
>            Assignee: Rob Davies
>            Priority: Blocker
>             Fix For: 5.3.0
>
>         Attachments: AMQ-2009Testcase2.zip, consumertest.zip, DispatchMultipleConsumersTest.java, JConsole-screenshot.jpg, testcase.zip
>
>
> Messages are not getting dispatched after a while (although it accepts new incoming messages) until restart of the broker. This problem is described in several posts.
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html
> There was also an issue opened in Spring project for this thinking it was Spring problem.
> http://jira.springframework.org/browse/SPR-5110
> I am not able to reproduce with Junit test case having BrokerService started with in the test case. I guess I am not hitting the right stress conditions this way. But when I run the test case against an externally running ActiveMQ instance backed with oracle database persistence, it is reproducible most of the times. This is not a every time failure situation, it takes more time once than the other.
> I was able to hit this situation of stuck messages on queue using following scenario most of the times:
> 1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer using cacheLevelName as CACHE_CONSUMER
> 2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance with 50 threads looping 20 times.
> 3) After a while, you will notice that Spring logs that no messages are being received but the messages are shown jconsole of ActiveMQ and the database backing it for persistence.
> But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove them from broker's database after acknowledge properly.
> Attached test case might help to reproduce when run against externally running stand alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER by sending messages to a queue with a camel route that moves messages from this queue to another and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3.
> Sorry about such a huge description but it is a real problem! A different team at our company are having this issue in production with 5.1. They are using it as an embedded broker with derby for persistence.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.