You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Gary Tully (JIRA)" <ji...@apache.org> on 2011/02/14 14:21:57 UTC
[jira] Resolved: (AMQ-3120) KahaDB error: "Could not locate data
file"
[ https://issues.apache.org/jira/browse/AMQ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully resolved AMQ-3120.
-----------------------------
Resolution: Fixed
fix committed in http://svn.apache.org/viewvc?rev=1070484&view=rev
interleaved checkpoint with journal add could cause inadvertent increment on the last update location, leaving current journal file as a gc candidate. resulting in the exception and hung destination.
Fix keeps lastUpdate independent of the current checkpoint.
> KahaDB error: "Could not locate data file"
> ------------------------------------------
>
> Key: AMQ-3120
> URL: https://issues.apache.org/jira/browse/AMQ-3120
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: linux: CentOS 5.5
> happens on both dual- and quad-cpu boxes, 4gb ram
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
> Reporter: Dan Checkoway
> Assignee: Gary Tully
> Fix For: 5.5.0
>
>
> I'm using an embedded broker (version 5.4.2) with persistence enabled. We're pumping hundreds of millions of messages per day through this thing. Every once in a while, all of a sudden the KahaDB directory starts growing uncontrollably, and these errors start spewing out in the log over and over...
> --------------------------
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; AbstractStoreCursor - Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue - Failed to page in more queue messages
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> ... 9 more
> Caused by: java.io.IOException: Could not locate data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> --------------------------
> It doesn't happen consistently, but it has happened numerous times. The only way to "fix" it is to kill the app, blow away the KahaDB data directory, and restart (ugh).
> Unfortunately this issue is not reproduceable.
> Here's the embedded broker config, fwiw:
> --------------------------
> <broker xmlns="http://activemq.apache.org/schema/core"
> id="embeddedActivemqBroker"
> useJmx="false"
> persistent="true"
> dataDirectory="/usr/local/embedded/activemq-data"
> schedulerSupport="false"
> advisorySupport="false"
> enableStatistics="true">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="20mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> </broker>
> --------------------------
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira