You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Pankaj Takawale (JIRA)" <ji...@apache.org> on 2014/12/01 23:48:12 UTC
[jira] [Updated] (AMQ-5459) leveldb failed to page in queue
messages
[ https://issues.apache.org/jira/browse/AMQ-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pankaj Takawale updated AMQ-5459:
---------------------------------
Description:
ActiveMQ Replica LevelDB cluster (3 nodes)
Zookeeper 3 nodes cluster
OS: CentOS 6.5
Size of messages: 50K to 500K
Number of messages sent: 9,500
Consumed: around 6000
I recycled master & slave at the same time while client was sending/consuming messages.
ActiveMQ cluster recovered fine, and client resumed producing/consuming messages.
After around two hours, Master faced following exceptions.
On these exceptions It demoted itself to a slave. New master got elected, and it faced the same issue. So master kept on rotating among cluster nodes.
I tried to restart ActiveMQ service on all three nodes. But activemq was facing same issue.
Is there any way to recover out of this state without loosing messages in queue?
Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch
java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue - cbtesta - Problem retrieving message for browse
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
was:
ActiveMQ Replica LevelDB cluster (3 nodes)
Zookeeper 3 nodes cluster
OS: CentOS 6.5
Size of messages: 50K to 500K
Number of messages sent: 9,500
Consumed: around 6000
I recycled master & slave at the same time while client was sending/consuming messages.
ActiveMQ cluster recovered fine, and client resumed producing/consuming messages.
After around two hours, Master faced following exceptions.
On these exceptions It demoted itself to a slave. New master got elected, and it faced the same issue.
I tried to restart ActiveMQ service on all three nodes. But activemq is facing same issue.
Is there any way to recover out of this state without loosing messages in queue?
Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch
java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue - cbtesta - Problem retrieving message for browse
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
> leveldb failed to page in queue messages
> ----------------------------------------
>
> Key: AMQ-5459
> URL: https://issues.apache.org/jira/browse/AMQ-5459
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: ActiveMQ Replica LevelDB cluster (3 nodes)
> Zookeeper 3 nodes cluster
> OS: CentOS 6.5
> Reporter: Pankaj Takawale
> Priority: Critical
>
> ActiveMQ Replica LevelDB cluster (3 nodes)
> Zookeeper 3 nodes cluster
> OS: CentOS 6.5
> Size of messages: 50K to 500K
> Number of messages sent: 9,500
> Consumed: around 6000
> I recycled master & slave at the same time while client was sending/consuming messages.
> ActiveMQ cluster recovered fine, and client resumed producing/consuming messages.
> After around two hours, Master faced following exceptions.
> On these exceptions It demoted itself to a slave. New master got elected, and it faced the same issue. So master kept on rotating among cluster nodes.
> I tried to restart ActiveMQ service on all three nodes. But activemq was facing same issue.
> Is there any way to recover out of this state without loosing messages in queue?
> Nov 25 15:03:35.777 [esta] Scheduler] - ERROR AbstractStoreCursor - cbtesta - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@72e4521c:testpt,batchResetNeeded=false,storeHasMessages=true,size=2516,cacheEnabled=false,maxBatchSize:200,hasSpace:true - Failed to fill batch
> java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:280)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:162)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
> at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
> at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
> Nov 25 15:03:35.777 [esta] Scheduler] - ERROR Queue - cbtesta - Problem retrieving message for browse
> java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:165)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1957)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2177)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1190)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:984)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue.access$100(Queue.java:104)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)[activemq-broker-5.10.0.jar:5.10.0]
> at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.10.0.jar:5.10.0]
> at java.util.TimerThread.mainLoop(Timer.java:555)[:1.7.0_71]
> at java.util.TimerThread.run(Timer.java:505)[:1.7.0_71]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)