You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mccraig_ <te...@gmail.com> on 2008/03/26 22:57:48 UTC
AMQMessageStore getting stuck
i'm getting exceptions from AMQMessageStore, and one of my queues is stuck
i put 200,000 messages on a queue, and after a couple of hours and 50,000
consumed messages consumers stopped getting any more messages. restarting
the broker gave me exceptions like below, but no resumption of delivery.
consumers on other queues are still getting messages delivered
this looks like https://issues.apache.org/activemq/browse/AMQ-1479, but i'm
running on a build of trunk, at revision 639434, and AMQ-1479 was fixed at
589314
there is no file "data-10" on my filesystem... "data-9", "data-12" and
"data-13" only
anyone else seen this ?
26 Mar 2008 20:29:07,262 DEBUG [TransportConnection] (ActiveMQ Transport:
tcp:///127.0.0.1:19017:) - Setting up new connection: /127.0.0.1:19017
26 Mar 2008 20:29:07,274 ERROR [AsyncDataManager] (ActiveMQ Transport:
tcp:///127.0.0.1:19017:) - Looking for key 10 but not found in fileMap:
{13=data-13 number = 13 , length = 14757778 refCount = 23, 9=data-9 number =
9 , length = 33554171 refCount = 0, 12=data-12 number = 12 , length =
33553108 refCount = 0}
26 Mar 2008 20:29:07,276 ERROR [AbstractStoreCursor] (ActiveMQ Transport:
tcp:///127.0.0.1:19017:) - Failed to fill batch
java.io.IOException: Failed to read to journal for: offset = 0, file = 10,
size = -1, type = 0. Reason: java.io.IOException: Could not locate data file
data--10
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:620)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:506)
at
org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:432)
at
org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:80)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:119)
at
org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:530)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:186)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:102)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:136)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1056)
at org.apache.activemq.broker.region.Queue.addSubscription(Queue.java:231)
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:264)
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:378)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:82)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:92)
at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:529)
at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:293)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:196)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:183)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Could not locate data file data--10
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:300)
at
org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:608)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:503)
... 28 more
--
View this message in context: http://www.nabble.com/AMQMessageStore-getting-stuck-tp16317366s2354p16317366.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: AMQMessageStore getting stuck
Posted by Rob Davies <ra...@gmail.com>.
On 26 Mar 2008, at 21:57, mccraig_ wrote:
>
> i'm getting exceptions from AMQMessageStore, and one of my queues is
> stuck
>
> i put 200,000 messages on a queue, and after a couple of hours and
> 50,000
> consumed messages consumers stopped getting any more messages.
> restarting
> the broker gave me exceptions like below, but no resumption of
> delivery.
> consumers on other queues are still getting messages delivered
>
> this looks like https://issues.apache.org/activemq/browse/AMQ-1479,
> but i'm
> running on a build of trunk, at revision 639434, and AMQ-1479 was
> fixed at
> 589314
>
> there is no file "data-10" on my filesystem... "data-9", "data-12" and
> "data-13" only
>
> anyone else seen this ?
>
> 26 Mar 2008 20:29:07,262 DEBUG [TransportConnection] (ActiveMQ
> Transport:
> tcp:///127.0.0.1:19017:) - Setting up new connection: /127.0.0.1:19017
> 26 Mar 2008 20:29:07,274 ERROR [AsyncDataManager] (ActiveMQ Transport:
> tcp:///127.0.0.1:19017:) - Looking for key 10 but not found in
> fileMap:
> {13=data-13 number = 13 , length = 14757778 refCount = 23, 9=data-9
> number =
> 9 , length = 33554171 refCount = 0, 12=data-12 number = 12 , length =
> 33553108 refCount = 0}
> 26 Mar 2008 20:29:07,276 ERROR [AbstractStoreCursor] (ActiveMQ
> Transport:
> tcp:///127.0.0.1:19017:) - Failed to fill batch
> java.io.IOException: Failed to read to journal for: offset = 0, file
> = 10,
> size = -1, type = 0. Reason: java.io.IOException: Could not locate
> data file
> data--10
> at
> org
> .apache
> .activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
> at
> org
> .apache
> .activemq
> .store
> .amq
> .AMQPersistenceAdapter
> .createReadException(AMQPersistenceAdapter.java:620)
> at
> org
> .apache
> .activemq
> .store
> .amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:506)
> at
> org
> .apache
> .activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:
> 432)
> at
> org
> .apache
> .activemq
> .store
> .amq
> .RecoveryListenerAdapter
> .recoverMessageReference(RecoveryListenerAdapter.java:54)
> at
> org
> .apache
> .activemq
> .store
> .kahadaptor
> .KahaReferenceStore.recoverReference(KahaReferenceStore.java:80)
> at
> org
> .apache
> .activemq
> .store
> .kahadaptor
> .KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:119)
> at
> org
> .apache
> .activemq
> .store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:
> 530)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:75)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:186)
> at
> org
> .apache
> .activemq
> .broker
> .region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
> 102)
> at
> org
> .apache
> .activemq
> .broker
> .region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:136)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1056)
> at
> org.apache.activemq.broker.region.Queue.addSubscription(Queue.java:
> 231)
> at
> org
> .apache
> .activemq
> .broker.region.AbstractRegion.addConsumer(AbstractRegion.java:264)
> at
> org
> .apache
> .activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:
> 378)
> at
> org
> .apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
> at
> org
> .apache
> .activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:82)
> at
> org
> .apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
> at
> org
> .apache
> .activemq
> .broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:92)
> at
> org
> .apache
> .activemq
> .broker
> .TransportConnection.processAddConsumer(TransportConnection.java:529)
> at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:
> 345)
> at
> org
> .apache
> .activemq
> .broker.TransportConnection.service(TransportConnection.java:293)
> at
> org.apache.activemq.broker.TransportConnection
> $1.onCommand(TransportConnection.java:181)
> at
> org
> .apache
> .activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> at
> org
> .apache
> .activemq
> .transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:
> 134)
> at
> org
> .apache
> .activemq
> .transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> at
> org
> .apache
> .activemq.transport.TransportSupport.doConsume(TransportSupport.java:
> 84)
> at
> org
> .apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:
> 196)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
> 183)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: java.io.IOException: Could not locate data file data--10
> at
> org
> .apache
> .activemq
> .kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:
> 300)
> at
> org
> .apache
> .activemq
> .kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:608)
> at
> org
> .apache
> .activemq
> .store
> .amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:503)
> ... 28 more
> --
> View this message in context: http://www.nabble.com/AMQMessageStore-getting-stuck-tp16317366s2354p16317366.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Do you think you could add a test case and open a jira ?
thanks,
Rob