You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Frank Gynnild (JIRA)" <ji...@apache.org> on 2011/08/24 11:16:55 UTC
[jira] [Created] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
----------------------------------------------------------------------------------------------------
Key: AMQ-3470
URL: https://issues.apache.org/jira/browse/AMQ-3470
Project: ActiveMQ
Issue Type: Bug
Environment: Used from .NET 4.0
Apache.NMS.ActiveMQ.dll - 1.5.1.2341
Apache.NMS.dll - 1.5.0.2194
ActiveMQ - 5.5.0
Reporter: Frank Gynnild
Priority: Blocker
We've a C# test client that constantly produces and consumes messages of various sizes.
After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
a certain state where:
- No more messages in the queue cannot be dequeued/received.
- Looking at the ActiveMQ browser, we're no longer able to see the message.
- The logs get filled up and rotate at a crazy pace.
- We aren't able to recover after this, restarting the ActiveMQ doesn't help.
We haven't been able to consistently reproduce this behavior, it can happen after
any numbers of messages (usually after some million messages). Last it took about 8
hours before we could come into this stage.
The tests are very basic.
I've been able to catch the log when this happens:
---
2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
... 7 more
2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
... 6 more
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
... 7 more
2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
... 7 more
2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
at org.apa
---
The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090148#comment-13090148 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
Also happened with the latest 5.6 snapshot unfortunately.
Happened after approximately 300 000 messages that got produced and consumed correctly.
---
2011-08-24 11:33:50,960 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@681e2ca7: startup date [Wed Aug 24 11:33:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2011-08-24 11:33:54,524 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
2011-08-24 11:33:54,541 | INFO | PListStore:[C:\ActiveMQ\data\IES_CARMINE\tmp_storage ] started | org.apache.activemq.store.kahadb.plist.PListStore | main
2011-08-24 11:33:54,642 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
2011-08-24 11:33:54,855 | INFO | ActiveMQ 5.6-SNAPSHOT JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
2011-08-24 11:33:54,855 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
2011-08-24 11:33:55,127 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2011-08-24 11:33:55,129 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2011-08-24 11:33:55,134 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-55091-1314178434891-0:1) started | org.apache.activemq.broker.BrokerService | main
2011-08-24 11:33:55,267 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
2011-08-24 11:33:56,278 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2011-08-24 11:33:56,656 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2011-08-24 11:33:57,055 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
2011-08-24 11:33:57,100 | INFO | WebApp@2005580923 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
2011-08-24 11:33:57,127 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
2011-08-24 13:27:49,774 | ERROR | Looking for key 652 but not found in fileMap: {653=db-653.log number = 653 , length = 1622} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-24 13:27:49,782 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@1d914a39:TestQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:813)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:959)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:489)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:478)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-24 13:27:49,793 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@1d914a39:TestQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:268)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:813)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:959)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:489)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:478)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
... 7 more
2011-08-24 13:27:49,799 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:268)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
... 6 more
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:813)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:959)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:489)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:478)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
... 7 more
2011-08-24 13:27:49,846 | ERROR | Looking for key 652 but not found in fileMap: {653=db-653.log number = 653 , length = 16999} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-24 13:27:49,846 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@1d914a39:TestQueue,batchResetNeeded=false,storeHasMessages=true,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:813)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:959)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:489)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:478)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-24 13:27:49,848 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@1d914a39:TestQueue,batchResetNeeded=false,storeHasMessages=true,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:268)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
at org.apache.kahadb.journal.Journal.read(Journal.java:597)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:813)
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:959)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:489)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:478)
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
... 7 more
2011-08-24 13:27:49,849 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1698)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1915)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1423)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-652.log
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:268)
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
... 6 more
---
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank Gynnild updated AMQ-3470:
-------------------------------
Affects Version/s: 5.5.0
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090104#comment-13090104 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
I will retest with the latest 5.6 as well to see if I can reproduce the behavior there.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Timothy Bish (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137365#comment-13137365 ]
Timothy Bish commented on AMQ-3470:
-----------------------------------
Any feedback on this, have you tested again using a later 5.6-SNAPSHOT? We are working hard to get 5.6 ready for release and would love to know if this one has been fixed.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091707#comment-13091707 ]
Gary Tully commented on AMQ-3470:
---------------------------------
to follow up on Dejan's comment, some detail on the MessageDatabase logging that will help understand the cleanup process. http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090163#comment-13090163 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
Unable to restart ActiveMQ after this occurs:
---
2011-08-24 14:19:48,446 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@6c4fc156: startup date [Wed Aug 24 14:19:48 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2011-08-24 14:19:52,644 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
2011-08-24 14:19:52,655 | INFO | PListStore:[C:\ActiveMQ\data\IES_CARMINE\tmp_storage ] started | org.apache.activemq.store.kahadb.plist.PListStore | main
2011-08-24 14:19:52,739 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
2011-08-24 14:19:53,162 | INFO | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-24 14:19:53,281 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-24 14:19:53,282 | INFO | Recovery replayed 1 operations from the journal in 0.092 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-24 14:19:53,463 | INFO | Some journal files are missing: [652] | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-24 14:19:53,469 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Detected missing/corrupt journal files. 1 messages affected. | org.apache.activemq.broker.BrokerService | main
java.io.IOException: Detected missing/corrupt journal files. 1 messages affected.
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:650)
at org.apache.activemq.store.kahadb.MessageDatabase$6.execute(MessageDatabase.java:473)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760)
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:471)
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:441)
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:303)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:361)
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:199)
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:167)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:167)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:494)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1536)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1409)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:574)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:106)
at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:106)
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:71)
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
at org.apache.activemq.console.Main.main(Main.java:107)
2011-08-24 14:19:53,482 | INFO | ActiveMQ Message Broker (IES_CARMINE, null) is shutting down | org.apache.activemq.broker.BrokerService | main
2011-08-24 14:19:53,483 | INFO | Connector openwire Stopped | org.apache.activemq.broker.TransportConnector | main
2011-08-24 14:19:53,494 | INFO | PListStore:[C:\ActiveMQ\data\IES_CARMINE\tmp_storage ] stopped | org.apache.activemq.store.kahadb.plist.PListStore | main
2011-08-24 14:19:53,495 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2011-08-24 14:19:53,495 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2011-08-24 14:19:53,495 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | main
2011-08-24 14:19:53,705 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, null) stopped | org.apache.activemq.broker.BrokerService | main
---
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090979#comment-13090979 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
Unfortunately, turning off the cache didn't help.Now it occurred again, after about 3Â 570Â 000 messages sent.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Timothy Bish (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3470.
-----------------------------
Resolution: Incomplete
No further input from the reporter regarding whether the current fixes have resolved the issue. Recommend testing using the latest 5.6-SNAPSHOT and the current NMS.ActiveMQ v1.5.3 release.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094530#comment-13094530 ]
Gary Tully commented on AMQ-3470:
---------------------------------
There is a small chance that the fix for https://issues.apache.org/jira/browse/AMQ-2868 may be relevant. In addition to the rework, our tracking of the lastupdate in the journal is simplified w.r.t to locking so it may have a positive effect.
Maybe do subsequent tests with tomorrows 5.6-SNAPSHOT.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091032#comment-13091032 ]
Dejan Bosanac commented on AMQ-3470:
------------------------------------
Thinking that rerunning your test with trace logging on org.apache.activemq.store.kahadb would also give us some clue why the journal file was picked for removal.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091004#comment-13091004 ]
Dejan Bosanac commented on AMQ-3470:
------------------------------------
The broker configuration would come handy as well
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091002#comment-13091002 ]
Gary Tully commented on AMQ-3470:
---------------------------------
thanks for the validation, unfortunately this means there must be an alternative cause. One last test may make sense, use the example producer/consumer ant tasks from a distro rather than your .net code. Details in the examples doc[1]. This will narrow the moving parts as we try and reproduce. What sort of hardware is the broker running on?
[1] http://activemq.apache.org/examples.html
Also, can you attach the the log file with the first exception. there may be some more context in there that can help narrow it down.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090178#comment-13090178 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
Gary: I will retest with
{code}
<policyEntry queue=">" useCache="false" />
{code}
thanks for the tip!
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091716#comment-13091716 ]
Gary Tully commented on AMQ-3470:
---------------------------------
one further thought for the moment. My suspicion now is the concurrentStoreAndDispatchQueues optimization, I think setting that to false may resolve this:{code}<persistenceAdapter>
<kahaDB ... concurrentStoreAndDispatchQueues="false"/>
</persistenceAdapter>{code}
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Frank Gynnild (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094472#comment-13094472 ]
Frank Gynnild commented on AMQ-3470:
------------------------------------
Thank you all for the ideas. I've been busy these last days, but I will pick up where I left and try the above suggestions. I'll keep you posted.
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3470) Unable to pick up messages anymore,
messages are lost - Looking for key xyz but not found in fileMap
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090157#comment-13090157 ]
Gary Tully commented on AMQ-3470:
---------------------------------
Thanks for the detail. We (dejanb and I) had a chat about this issue earlier and may have found the root cause. If our thoughts are correct, disabling the cache, via xml configuration, should work around the issue till we get a fix.
{code}<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" useCache="false" />
</policyEntries>
</policyMap>
</destinationPolicy>{code}
> Unable to pick up messages anymore, messages are lost - Looking for key xyz but not found in fileMap
> ----------------------------------------------------------------------------------------------------
>
> Key: AMQ-3470
> URL: https://issues.apache.org/jira/browse/AMQ-3470
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.5.0
> Environment: Used from .NET 4.0
> Apache.NMS.ActiveMQ.dll - 1.5.1.2341
> Apache.NMS.dll - 1.5.0.2194
> ActiveMQ - 5.5.0
> Reporter: Frank Gynnild
> Priority: Blocker
>
> We've a C# test client that constantly produces and consumes messages of various sizes.
> After successfully sending and receiving about 5.5 million messages, ActiveMQ enters
> a certain state where:
> - No more messages in the queue cannot be dequeued/received.
> - Looking at the ActiveMQ browser, we're no longer able to see the message.
> - The logs get filled up and rotate at a crazy pace.
> - We aren't able to recover after this, restarting the ActiveMQ doesn't help.
> We haven't been able to consistently reproduce this behavior, it can happen after
> any numbers of messages (usually after some million messages). Last it took about 8
> hours before we could come into this stage.
> The tests are very basic.
> I've been able to catch the log when this happens:
> ---
> 2011-08-22 11:13:50,919 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3479e304: startup date [Mon Aug 22 11:13:50 CEST 2011]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2011-08-22 11:13:54,638 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
> 2011-08-22 11:13:54,645 | INFO | PListStore:C:\ActiveMQ\data\IES_CARMINE\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2011-08-22 11:13:54,723 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\ActiveMQ\data\kahadb] | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,932 | INFO | ActiveMQ 5.5.0 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:54,933 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,224 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2011-08-22 11:13:55,226 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
> 2011-08-22 11:13:55,233 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-60118-1314004434990-0:1) started | org.apache.activemq.broker.BrokerService | main
> 2011-08-22 11:13:55,390 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,043 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
> 2011-08-22 11:13:56,367 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
> 2011-08-22 11:13:56,807 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,852 | INFO | WebApp@280236390 at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
> 2011-08-22 11:13:56,876 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
> 2011-08-23 19:45:41,881 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 1949} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,935 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,939 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,940 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> ... 6 more
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,942 | ERROR | Looking for key 11856 but not found in fileMap: {11857=db-11857.log number = 11857 , length = 19540} | org.apache.kahadb.journal.Journal | Queue:TestQueue
> 2011-08-23 19:45:41,943 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> 2011-08-23 19:45:41,944 | ERROR | QueueStorePrefetch314000409 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
> java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
> at org.apache.kahadb.journal.Journal.read(Journal.java:597)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)
> ... 7 more
> 2011-08-23 19:45:41,945 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1712)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1932)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1440)
> at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:104)
> at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\ActiveMQ\data\kahadb\db-11856.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)
> at org.apa
> ---
> The logs repeat in an endless matter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira