You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by gtamir <gi...@active.com> on 2009/02/11 23:51:22 UTC
Re: 5.2.0 dequeue counter=0 pending queue stalls
Encountering similar situation. Found it was sufficient to just delete the
kaha store to cause listeners to start picking up messages (stop ActiveMQ
5.2 final, delete data/kr-store and restart ActiveMQ).
Switched to using Derby as a JDBC data store instead of the kaha store - no
improvement.
thomaskwan wrote:
>
> We are using ActiveMQ 5.2.0 the latest stable release. This is what i have
> seen:
>
> Client ID: ...
> Subscription Name: ...
> Connection ID: ...
> Pending Queue Size: 9894
> Dispatched Queue Size 0
> Dispatched Counter: 0
> Enqueue Counter: 9894
> Dequeue Counter: 0
>
> Within the same MQ server, some consumers work, some just got stuck into
> the state that I have
> mentioned above. The active MQ log shows:
>
>
> 2009-01-25 14:33:14,489 [168.20.36:53300] ERROR RecoveryListenerAdapter
> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
> recovered from the data store - already dispatched
> 2009-01-25 14:33:14,490 [168.20.36:53300] ERROR RecoveryListenerAdapter
> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
> recovered from the data store - already dispatched
> ...
> 2009-01-25 14:33:15,428 [168.20.36:53310] ERROR DataManagerImpl
> - Looking for key 4 but not found in fileMap: {5=data-topic-data-5 number
> = 5 , length = 2097120 refCount = 2460, 6=data-topic-data-6 number = 6 ,
> length = 2097135 refCount = 2942, 7=data-topic-data-7 number = 7 , length
> = 2097127 refCount = 2844, 8=data-topic-data-8 number = 8 , length =
> 2097111 refCount = 3308, 9=data-topic-data-9 number = 9 , length = 2097127
> refCount = 3196, 10=data-topic-data-10 number = 10 , length = 2097111
> refCount = 3341, 11=data-topic-data-11 number = 11 , length = 2097127
> refCount = 3625, 12=data-topic-data-12 number = 12 , length = 2097111
> refCount = 3602, 13=data-topic-data-13 number = 13 , length = 2097127
> refCount = 3544, 14=data-topic-data-14 number = 14 , length = 2097111
> refCount = 3412, 15=data-topic-data-15 number = 15 , length = 2097127
> refCount = 3698, 17=data-topic-data-17 number = 17 , length = 2097127
> refCount = 3612, 16=data-topic-data-16 number = 16 , length = 2097111
> refCount = 3570, 19=data-topic-data-19 number = 19 , length = 2097127
> refCount = 3588, 18=data-topic-data-18 number = 18 , length = 2097111
> refCount = 3468, 21=data-topic-data-21 number = 21 , length = 2097127
> refCount = 3322, 20=data-topic-data-20 number = 20 , length = 2097111
> refCount = 3284, 23=data-topic-data-23 number = 23 , length = 2097127
> refCount = 3336, 22=data-topic-data-22 number = 22 , length = 2097111
> refCount = 3188, 25=data-topic-data-25 number = 25 , length = 2097127
> refCount = 3270, 24=data-topic-data-24 number = 24 , length = 2097111
> refCount = 3322, 27=data-topic-data-27 number = 27 , length = 2097127
> refCount = 3240, 26=data-topic-data-26 number = 26 , length = 2097111
> refCount = 3214, 29=data-topic-data-29 number = 29 , length = 2097127
> refCount = 3290, 28=data-topic-data-28 number = 28 , length = 2097111
> refCount = 3126, 31=data-topic-data-31 number = 31 , length = 2097127
> refCount = 3382, 30=data-topic-data-30 number = 30 , length = 2097111
> refCount = 3200, 34=data-topic-data-34 number = 34 , length = 2097111
> refCount = 2928, 35=data-topic-data-35 number = 35 , length = 2097127
> refCount = 3094, 32=data-topic-data-32 number = 32 , length = 2097111
> refCount = 2866, 33=data-topic-data-33 number = 33 , length = 2097127
> refCount = 3096, 38=data-topic-data-38 number = 38 , length = 2097111
> refCount = 3202, 39=data-topic-data-39 number = 39 , length = 2097127
> refCount = 3158, 36=data-topic-data-36 number = 36 , length = 2097111
> refCount = 3146, 37=data-topic-data-37 number = 37 , length = 2097127
> refCount = 3082, 42=data-topic-data-42 number = 42 , length = 2097111
> refCount = 9917, 43=data-topic-data-43 number = 43 , length = 2097127
> refCount = 24897, 40=data-topic-data-40 number = 40 , length = 2097111
> refCount = 3144, 41=data-topic-data-41 number = 41 , length = 2097127
> refCount = 3344, 44=data-topic-data-44 number = 44 , length = 2097111
> refCount = 24650, 45=data-topic-data-45 number = 45 , length = 1051587
> refCount = 12466}
> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR MapContainerImpl
> - Failed to get value for offset=17748, key=(4, 610628, 47), value=(4,
> 610680, 63), previousItem=102, nextItem=-1
> java.io.IOException: Could not locate data file data-topic-data-4
> at
> org.apache.activemq.kaha.impl.data.DataManagerImpl.getDataFile(DataManagerImpl.java:129)
> at
> org.apache.activemq.kaha.impl.data.SyncDataFileReader.readItem(SyncDataFileReader.java:65)
> at
> org.apache.activemq.kaha.impl.data.DataManagerImpl.readItem(DataManagerImpl.java:141)
> at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:481)
> at
> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
> at
> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
> at
> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
> ..
> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR AbstractStoreCursor
> - Failed to fill batch
> org.apache.activemq.kaha.RuntimeStoreException: java.io.IOException: Could
> not locate data file data-topic-data-4
> at
> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:484)
> at
> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
> at
> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
> at
> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:227)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:100)
> at
> org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.reset(StoreDurableSubscriberCursor.java:233)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:566)
> at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> at
> org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:197)
> at
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> at
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> at
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> at org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> ..
>
> To fix this problem, we have to delete the data store and let MQ recreate
> it. My question is
> that is there a way to recover from this situation? More importantly, is
> there a way to
> prevent ourself to get into this situation.
>
> thanks
> thomas
>
--
View this message in context: http://www.nabble.com/5.2.0-dequeue-counter%3D0-pending-queue-stalls-tp21658012p21966159.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: 5.2.0 dequeue counter=0 pending queue stalls
Posted by gtamir <gi...@active.com>.
Configuration: ActiveMQ 5.2 final running on Centos 5 ( 2.6.18-92.1.6.el5 #1
SMP Wed Jun 25 13:49:24 EDT 2008 i686 i686 i386 GNU/Linux) VM with 4
available processors and 3GB of RAM.
- 22 clients (listeners) are running in the same VM, listening to 2
queues.
- Messages are put on the queues by a single process at a fairly high
rate - at least 20 a second. Messages are fairly small - usually less then
10K each.
- The clients reply to temporary queues help by a single process.
Reply messages can range in size from 100b to 14MB. To save on the cost of
creating and destroying temporary queues, 20 temporary queues are pooled and
used to hold replies.
- Selectors are used by all clients to determine who should peek up
specific messages.
- 99% of messages are NON_PERSISTENT.
elihusmails wrote:
>
> I ran into the same problem. Could you describe your configuration.
>
>
>
> On Wed, Feb 11, 2009 at 5:51 PM, gtamir <gi...@active.com> wrote:
>>
>> Encountering similar situation. Found it was sufficient to just delete
>> the
>> kaha store to cause listeners to start picking up messages (stop ActiveMQ
>> 5.2 final, delete data/kr-store and restart ActiveMQ).
>>
>> Switched to using Derby as a JDBC data store instead of the kaha store -
>> no
>> improvement.
>>
>>
>> thomaskwan wrote:
>>>
>>> We are using ActiveMQ 5.2.0 the latest stable release. This is what i
>>> have
>>> seen:
>>>
>>> Client ID: ...
>>> Subscription Name: ...
>>> Connection ID: ...
>>> Pending Queue Size: 9894
>>> Dispatched Queue Size 0
>>> Dispatched Counter: 0
>>> Enqueue Counter: 9894
>>> Dequeue Counter: 0
>>>
>>> Within the same MQ server, some consumers work, some just got stuck into
>>> the state that I have
>>> mentioned above. The active MQ log shows:
>>>
>>>
>>> 2009-01-25 14:33:14,489 [168.20.36:53300] ERROR RecoveryListenerAdapter
>>> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
>>> recovered from the data store - already dispatched
>>> 2009-01-25 14:33:14,490 [168.20.36:53300] ERROR RecoveryListenerAdapter
>>> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
>>> recovered from the data store - already dispatched
>>> ...
>>> 2009-01-25 14:33:15,428 [168.20.36:53310] ERROR DataManagerImpl
>>> - Looking for key 4 but not found in fileMap: {5=data-topic-data-5
>>> number
>>> = 5 , length = 2097120 refCount = 2460, 6=data-topic-data-6 number = 6 ,
>>> length = 2097135 refCount = 2942, 7=data-topic-data-7 number = 7 ,
>>> length
>>> = 2097127 refCount = 2844, 8=data-topic-data-8 number = 8 , length =
>>> 2097111 refCount = 3308, 9=data-topic-data-9 number = 9 , length =
>>> 2097127
>>> refCount = 3196, 10=data-topic-data-10 number = 10 , length = 2097111
>>> refCount = 3341, 11=data-topic-data-11 number = 11 , length = 2097127
>>> refCount = 3625, 12=data-topic-data-12 number = 12 , length = 2097111
>>> refCount = 3602, 13=data-topic-data-13 number = 13 , length = 2097127
>>> refCount = 3544, 14=data-topic-data-14 number = 14 , length = 2097111
>>> refCount = 3412, 15=data-topic-data-15 number = 15 , length = 2097127
>>> refCount = 3698, 17=data-topic-data-17 number = 17 , length = 2097127
>>> refCount = 3612, 16=data-topic-data-16 number = 16 , length = 2097111
>>> refCount = 3570, 19=data-topic-data-19 number = 19 , length = 2097127
>>> refCount = 3588, 18=data-topic-data-18 number = 18 , length = 2097111
>>> refCount = 3468, 21=data-topic-data-21 number = 21 , length = 2097127
>>> refCount = 3322, 20=data-topic-data-20 number = 20 , length = 2097111
>>> refCount = 3284, 23=data-topic-data-23 number = 23 , length = 2097127
>>> refCount = 3336, 22=data-topic-data-22 number = 22 , length = 2097111
>>> refCount = 3188, 25=data-topic-data-25 number = 25 , length = 2097127
>>> refCount = 3270, 24=data-topic-data-24 number = 24 , length = 2097111
>>> refCount = 3322, 27=data-topic-data-27 number = 27 , length = 2097127
>>> refCount = 3240, 26=data-topic-data-26 number = 26 , length = 2097111
>>> refCount = 3214, 29=data-topic-data-29 number = 29 , length = 2097127
>>> refCount = 3290, 28=data-topic-data-28 number = 28 , length = 2097111
>>> refCount = 3126, 31=data-topic-data-31 number = 31 , length = 2097127
>>> refCount = 3382, 30=data-topic-data-30 number = 30 , length = 2097111
>>> refCount = 3200, 34=data-topic-data-34 number = 34 , length = 2097111
>>> refCount = 2928, 35=data-topic-data-35 number = 35 , length = 2097127
>>> refCount = 3094, 32=data-topic-data-32 number = 32 , length = 2097111
>>> refCount = 2866, 33=data-topic-data-33 number = 33 , length = 2097127
>>> refCount = 3096, 38=data-topic-data-38 number = 38 , length = 2097111
>>> refCount = 3202, 39=data-topic-data-39 number = 39 , length = 2097127
>>> refCount = 3158, 36=data-topic-data-36 number = 36 , length = 2097111
>>> refCount = 3146, 37=data-topic-data-37 number = 37 , length = 2097127
>>> refCount = 3082, 42=data-topic-data-42 number = 42 , length = 2097111
>>> refCount = 9917, 43=data-topic-data-43 number = 43 , length = 2097127
>>> refCount = 24897, 40=data-topic-data-40 number = 40 , length = 2097111
>>> refCount = 3144, 41=data-topic-data-41 number = 41 , length = 2097127
>>> refCount = 3344, 44=data-topic-data-44 number = 44 , length = 2097111
>>> refCount = 24650, 45=data-topic-data-45 number = 45 , length = 1051587
>>> refCount = 12466}
>>> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR MapContainerImpl
>>> - Failed to get value for offset=17748, key=(4, 610628, 47), value=(4,
>>> 610680, 63), previousItem=102, nextItem=-1
>>> java.io.IOException: Could not locate data file data-topic-data-4
>>> at
>>> org.apache.activemq.kaha.impl.data.DataManagerImpl.getDataFile(DataManagerImpl.java:129)
>>> at
>>> org.apache.activemq.kaha.impl.data.SyncDataFileReader.readItem(SyncDataFileReader.java:65)
>>> at
>>> org.apache.activemq.kaha.impl.data.DataManagerImpl.readItem(DataManagerImpl.java:141)
>>> at
>>> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:481)
>>> at
>>> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
>>> at
>>> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
>>> at
>>> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
>>> ..
>>> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR AbstractStoreCursor
>>> - Failed to fill batch
>>> org.apache.activemq.kaha.RuntimeStoreException: java.io.IOException:
>>> Could
>>> not locate data file data-topic-data-4
>>> at
>>> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:484)
>>> at
>>> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
>>> at
>>> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
>>> at
>>> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
>>> at
>>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:227)
>>> at
>>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:100)
>>> at
>>> org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.reset(StoreDurableSubscriberCursor.java:233)
>>> at
>>> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:566)
>>> at
>>> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
>>> at
>>> org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:197)
>>> at
>>> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
>>> at
>>> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
>>> at
>>> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
>>> at org.apache.activemq.broker.region.Topic.send(Topic.java:365)
>>> at
>>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
>>> at
>>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
>>> at
>>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>>> at
>>> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>>> at
>>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>>> at
>>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>>> at
>>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
>>> at
>>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>>> at
>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
>>> ..
>>>
>>> To fix this problem, we have to delete the data store and let MQ
>>> recreate
>>> it. My question is
>>> that is there a way to recover from this situation? More importantly, is
>>> there a way to
>>> prevent ourself to get into this situation.
>>>
>>> thanks
>>> thomas
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/5.2.0-dequeue-counter%3D0-pending-queue-stalls-tp21658012p21966159.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/5.2.0-dequeue-counter%3D0-pending-queue-stalls-tp21658012p22211339.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: 5.2.0 dequeue counter=0 pending queue stalls
Posted by Mark Webb <el...@gmail.com>.
I ran into the same problem. Could you describe your configuration.
On Wed, Feb 11, 2009 at 5:51 PM, gtamir <gi...@active.com> wrote:
>
> Encountering similar situation. Found it was sufficient to just delete the
> kaha store to cause listeners to start picking up messages (stop ActiveMQ
> 5.2 final, delete data/kr-store and restart ActiveMQ).
>
> Switched to using Derby as a JDBC data store instead of the kaha store - no
> improvement.
>
>
> thomaskwan wrote:
>>
>> We are using ActiveMQ 5.2.0 the latest stable release. This is what i have
>> seen:
>>
>> Client ID: ...
>> Subscription Name: ...
>> Connection ID: ...
>> Pending Queue Size: 9894
>> Dispatched Queue Size 0
>> Dispatched Counter: 0
>> Enqueue Counter: 9894
>> Dequeue Counter: 0
>>
>> Within the same MQ server, some consumers work, some just got stuck into
>> the state that I have
>> mentioned above. The active MQ log shows:
>>
>>
>> 2009-01-25 14:33:14,489 [168.20.36:53300] ERROR RecoveryListenerAdapter
>> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
>> recovered from the data store - already dispatched
>> 2009-01-25 14:33:14,490 [168.20.36:53300] ERROR RecoveryListenerAdapter
>> - Message id ID:hefeed1-48611-1232757655348-4:140816:-1:1:1 could not be
>> recovered from the data store - already dispatched
>> ...
>> 2009-01-25 14:33:15,428 [168.20.36:53310] ERROR DataManagerImpl
>> - Looking for key 4 but not found in fileMap: {5=data-topic-data-5 number
>> = 5 , length = 2097120 refCount = 2460, 6=data-topic-data-6 number = 6 ,
>> length = 2097135 refCount = 2942, 7=data-topic-data-7 number = 7 , length
>> = 2097127 refCount = 2844, 8=data-topic-data-8 number = 8 , length =
>> 2097111 refCount = 3308, 9=data-topic-data-9 number = 9 , length = 2097127
>> refCount = 3196, 10=data-topic-data-10 number = 10 , length = 2097111
>> refCount = 3341, 11=data-topic-data-11 number = 11 , length = 2097127
>> refCount = 3625, 12=data-topic-data-12 number = 12 , length = 2097111
>> refCount = 3602, 13=data-topic-data-13 number = 13 , length = 2097127
>> refCount = 3544, 14=data-topic-data-14 number = 14 , length = 2097111
>> refCount = 3412, 15=data-topic-data-15 number = 15 , length = 2097127
>> refCount = 3698, 17=data-topic-data-17 number = 17 , length = 2097127
>> refCount = 3612, 16=data-topic-data-16 number = 16 , length = 2097111
>> refCount = 3570, 19=data-topic-data-19 number = 19 , length = 2097127
>> refCount = 3588, 18=data-topic-data-18 number = 18 , length = 2097111
>> refCount = 3468, 21=data-topic-data-21 number = 21 , length = 2097127
>> refCount = 3322, 20=data-topic-data-20 number = 20 , length = 2097111
>> refCount = 3284, 23=data-topic-data-23 number = 23 , length = 2097127
>> refCount = 3336, 22=data-topic-data-22 number = 22 , length = 2097111
>> refCount = 3188, 25=data-topic-data-25 number = 25 , length = 2097127
>> refCount = 3270, 24=data-topic-data-24 number = 24 , length = 2097111
>> refCount = 3322, 27=data-topic-data-27 number = 27 , length = 2097127
>> refCount = 3240, 26=data-topic-data-26 number = 26 , length = 2097111
>> refCount = 3214, 29=data-topic-data-29 number = 29 , length = 2097127
>> refCount = 3290, 28=data-topic-data-28 number = 28 , length = 2097111
>> refCount = 3126, 31=data-topic-data-31 number = 31 , length = 2097127
>> refCount = 3382, 30=data-topic-data-30 number = 30 , length = 2097111
>> refCount = 3200, 34=data-topic-data-34 number = 34 , length = 2097111
>> refCount = 2928, 35=data-topic-data-35 number = 35 , length = 2097127
>> refCount = 3094, 32=data-topic-data-32 number = 32 , length = 2097111
>> refCount = 2866, 33=data-topic-data-33 number = 33 , length = 2097127
>> refCount = 3096, 38=data-topic-data-38 number = 38 , length = 2097111
>> refCount = 3202, 39=data-topic-data-39 number = 39 , length = 2097127
>> refCount = 3158, 36=data-topic-data-36 number = 36 , length = 2097111
>> refCount = 3146, 37=data-topic-data-37 number = 37 , length = 2097127
>> refCount = 3082, 42=data-topic-data-42 number = 42 , length = 2097111
>> refCount = 9917, 43=data-topic-data-43 number = 43 , length = 2097127
>> refCount = 24897, 40=data-topic-data-40 number = 40 , length = 2097111
>> refCount = 3144, 41=data-topic-data-41 number = 41 , length = 2097127
>> refCount = 3344, 44=data-topic-data-44 number = 44 , length = 2097111
>> refCount = 24650, 45=data-topic-data-45 number = 45 , length = 1051587
>> refCount = 12466}
>> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR MapContainerImpl
>> - Failed to get value for offset=17748, key=(4, 610628, 47), value=(4,
>> 610680, 63), previousItem=102, nextItem=-1
>> java.io.IOException: Could not locate data file data-topic-data-4
>> at
>> org.apache.activemq.kaha.impl.data.DataManagerImpl.getDataFile(DataManagerImpl.java:129)
>> at
>> org.apache.activemq.kaha.impl.data.SyncDataFileReader.readItem(SyncDataFileReader.java:65)
>> at
>> org.apache.activemq.kaha.impl.data.DataManagerImpl.readItem(DataManagerImpl.java:141)
>> at
>> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:481)
>> at
>> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
>> at
>> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
>> at
>> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
>> ..
>> 2009-01-25 14:33:15,429 [168.20.36:53310] ERROR AbstractStoreCursor
>> - Failed to fill batch
>> org.apache.activemq.kaha.RuntimeStoreException: java.io.IOException: Could
>> not locate data file data-topic-data-4
>> at
>> org.apache.activemq.kaha.impl.container.MapContainerImpl.getValue(MapContainerImpl.java:484)
>> at
>> org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:254)
>> at
>> org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:59)
>> at
>> org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:94)
>> at
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:227)
>> at
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:100)
>> at
>> org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.reset(StoreDurableSubscriberCursor.java:233)
>> at
>> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:566)
>> at
>> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
>> at
>> org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:197)
>> at
>> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
>> at
>> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
>> at
>> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
>> at org.apache.activemq.broker.region.Topic.send(Topic.java:365)
>> at
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
>> at
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
>> at
>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>> at
>> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>> at
>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>> at
>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>> at
>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
>> at
>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>> at
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
>> ..
>>
>> To fix this problem, we have to delete the data store and let MQ recreate
>> it. My question is
>> that is there a way to recover from this situation? More importantly, is
>> there a way to
>> prevent ourself to get into this situation.
>>
>> thanks
>> thomas
>>
>
> --
> View this message in context: http://www.nabble.com/5.2.0-dequeue-counter%3D0-pending-queue-stalls-tp21658012p21966159.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>