You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by dcheckoway <dc...@gmail.com> on 2011/01/06 13:39:35 UTC
KahaDB error: "Could not locate data file"
I'm using an embedded broker (version 5.4.2) with persistence enabled. We're
pumping hundreds of millions of messages per day through this thing. Every
once in a while, all of a sudden the KahaDB directory starts growing
uncontrollably, and these errors start spewing out in the log over and
over...
--------------------------
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
AbstractStoreCursor - Failed to fill batch
java.lang.RuntimeException: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
Failed to page in more queue messages
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
... 9 more
Caused by: java.io.IOException: Could not locate data file
/usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
at org.apache.kahadb.journal.Journal.read(Journal.java:592)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
... 10 more
--------------------------
It doesn't happen consistently, but it has happened numerous times. The
only way to "fix" it is to kill the app, blow away the KahaDB data
directory, and restart (ugh).
Has anybody else seen this issue? Any idea why it happens?
Here's the embedded broker config, fwiw:
--------------------------
<broker xmlns="http://activemq.apache.org/schema/core"
id="embeddedActivemqBroker"
useJmx="false"
persistent="true"
dataDirectory="/usr/local/embedded/activemq-data"
schedulerSupport="false"
advisorySupport="false"
enableStatistics="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="20mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
</broker>
--------------------------
Help?!
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDB error: "Could not locate data file"
Posted by Norman Maurer <no...@apache.org>.
In 5.4.2 its kahadb..
Bye,
Norman
2011/1/6 dcheckoway <dc...@gmail.com>:
>
> Yeah, we're using whatever is the default persistence provider in 5.4.2. I
> opened a fresh ticket for it:
>
> https://issues.apache.org/jira/browse/AMQ-3120
>
> Thanks,
> Dan
>
> On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] <
> ml-node+3177622-1458695149-140184@n4.nabble.com<ml...@n4.nabble.com>
>> wrote:
>
>> AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
>> to be different from your use case?
>>
>> On 6 January 2011 15:14, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>>
>> wrote:
>>
>> >
>> > Gary,
>> >
>> > Thanks for the reply and advice. I'll definitely try disabling
>> > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an
>> IO
>> > exception handler.
>> >
>> > Unfortunately the issue can't reliably be reproduced. We have a cluster
>> of
>> > servers all running their own local embedded queue, and it happens on one
>>
>> > here, one there.
>> >
>> > That said, somebody else already opened a JIRA ticket for what looks like
>> a
>> > similar/nearly identical issue:
>> > https://issues.apache.org/jira/browse/AMQ-3051
>> >
>> > Dan
>> >
>> >
>> > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
>> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden
>> email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>>
>> >> wrote:
>> >
>> >> One thought, there may be an issue with the default setting of true
>> >> for concurrentStoreAndDispatchQueues that is causing the index to be
>> >> out of sync with the store.
>> >>
>> >> It is worth trying to disable that, use:
>> >> <persistenceAdapter>
>> >> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
>> >> concurrentStoreAndDispatchQueues="false"/>
>> >> </persistenceAdapter>
>> >>
>> >> How often is this occurring, to really get to the bottom of this a
>> >> test case will be needed.
>> >> Does that data file eventually get written, or are there any disk space
>> >> issues?
>> >> It may help to configure an IO exception handler just in case.
>> >> http://activemq.apache.org/configurable-ioexception-handling.html
>> >>
>> >> Would be a good idea to create a jira issue to track this and attach
>> >> what ever log and application information you have to it.
>> >>
>> >> On 6 January 2011 12:39, dcheckoway <[hidden email]<
>> http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
>> >> wrote:
>> >>
>> >> >
>> >> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>>
>> >> We're
>> >> > pumping hundreds of millions of messages per day through this thing.
>> >> Every
>> >> > once in a while, all of a sudden the KahaDB directory starts growing
>> >> > uncontrollably, and these errors start spewing out in the log over and
>>
>> >> > over...
>> >> >
>> >> > --------------------------
>> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> >> > AbstractStoreCursor - Failed to fill batch
>> >> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>>
>> >> file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >>
>> >> > at
>> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >> > at
>> >> >
>> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >> > at
>> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >>
>> >> > at java.lang.Thread.run(Thread.java:619)
>> >> > Caused by: java.io.IOException: Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> > at
>> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>> >>
>> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >>
>> >> > at
>> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >>
>> >> > ... 10 more
>> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> Queue
>> >> -
>> >> > Failed to page in more queue messages
>> >> > java.lang.RuntimeException: java.lang.RuntimeException:
>> >> java.io.IOException:
>> >> > Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> >>
>> >> > at
>> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> >> > at
>> >> >
>> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> >> > at
>> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> >>
>> >> > at java.lang.Thread.run(Thread.java:619)
>> >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
>> >> locate
>> >> > data file
>> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> >>
>> >> > ... 9 more
>> >> > Caused by: java.io.IOException: Could not locate data file
>> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> >> > at
>> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>> >>
>> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> >>
>> >> > at
>> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> >>
>> >> > at
>> >> >
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> >>
>> >> > ... 10 more
>> >> > --------------------------
>> >> >
>> >> > It doesn't happen consistently, but it has happened numerous times.
>> The
>> >> > only way to "fix" it is to kill the app, blow away the KahaDB data
>> >> > directory, and restart (ugh).
>> >> >
>> >> > Has anybody else seen this issue? Any idea why it happens?
>> >> >
>> >> > Here's the embedded broker config, fwiw:
>> >> > --------------------------
>> >> > <broker xmlns="http://activemq.apache.org/schema/core"
>> >> > id="embeddedActivemqBroker"
>> >> > useJmx="false"
>> >> > persistent="true"
>> >> > dataDirectory="/usr/local/embedded/activemq-data"
>> >> > schedulerSupport="false"
>> >> > advisorySupport="false"
>> >> > enableStatistics="true">
>> >> > <destinationPolicy>
>> >> > <policyMap>
>> >> > <policyEntries>
>> >> > <policyEntry queue=">" producerFlowControl="false"
>> >> > memoryLimit="20mb"/>
>> >> > </policyEntries>
>> >> > </policyMap>
>> >> > </destinationPolicy>
>> >> > </broker>
>> >> > --------------------------
>> >> >
>> >> > Help?!
>> >> >
>> >> > -----
>> >> > Dan Checkoway
>> >> > dcheckoway gmail com
>> >> > --
>> >> > View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
>> <
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>>
>>
>> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> http://blog.garytully.com
>> >> http://fusesource.com
>> >>
>> >>
>> >> ------------------------------
>> >> View message @
>> >>
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t>
>> >>
>> >>
>> >>
>> >
>> >
>> > -----
>> > Dan Checkoway
>> > dcheckoway gmail com
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t>
>>
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>> http://fusesource.com
>>
>>
>> ------------------------------
>> View message @
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html
>>
>> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>>
>>
>
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: KahaDB error: "Could not locate data file"
Posted by dcheckoway <dc...@gmail.com>.
Yeah, we're using whatever is the default persistence provider in 5.4.2. I
opened a fresh ticket for it:
https://issues.apache.org/jira/browse/AMQ-3120
Thanks,
Dan
On Thu, Jan 6, 2011 at 10:42 AM, Gary Tully [via ActiveMQ] <
ml-node+3177622-1458695149-140184@n4.nabble.com<ml...@n4.nabble.com>
> wrote:
> AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
> to be different from your use case?
>
> On 6 January 2011 15:14, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177622&i=0>>
> wrote:
>
> >
> > Gary,
> >
> > Thanks for the reply and advice. I'll definitely try disabling
> > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an
> IO
> > exception handler.
> >
> > Unfortunately the issue can't reliably be reproduced. We have a cluster
> of
> > servers all running their own local embedded queue, and it happens on one
>
> > here, one there.
> >
> > That said, somebody else already opened a JIRA ticket for what looks like
> a
> > similar/nearly identical issue:
> > https://issues.apache.org/jira/browse/AMQ-3051
> >
> > Dan
> >
> >
> > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3177622&i=1><[hidden
> email] <http://user/SendEmail.jtp?type=node&node=3177622&i=2>>
> >> wrote:
> >
> >> One thought, there may be an issue with the default setting of true
> >> for concurrentStoreAndDispatchQueues that is causing the index to be
> >> out of sync with the store.
> >>
> >> It is worth trying to disable that, use:
> >> <persistenceAdapter>
> >> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
> >> concurrentStoreAndDispatchQueues="false"/>
> >> </persistenceAdapter>
> >>
> >> How often is this occurring, to really get to the bottom of this a
> >> test case will be needed.
> >> Does that data file eventually get written, or are there any disk space
> >> issues?
> >> It may help to configure an IO exception handler just in case.
> >> http://activemq.apache.org/configurable-ioexception-handling.html
> >>
> >> Would be a good idea to create a jira issue to track this and attach
> >> what ever log and application information you have to it.
> >>
> >> On 6 January 2011 12:39, dcheckoway <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
> >> wrote:
> >>
> >> >
> >> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>
> >> We're
> >> > pumping hundreds of millions of messages per day through this thing.
> >> Every
> >> > once in a while, all of a sudden the KahaDB directory starts growing
> >> > uncontrollably, and these errors start spewing out in the log over and
>
> >> > over...
> >> >
> >> > --------------------------
> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> >> > AbstractStoreCursor - Failed to fill batch
> >> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>
> >> file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> >>
> >> > at
> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> >> > at
> >> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> >> > at
> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> >> > at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> >>
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> >>
> >> > at java.lang.Thread.run(Thread.java:619)
> >> > Caused by: java.io.IOException: Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> > at
> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> >>
> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> >>
> >> > at
> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> >>
> >> > ... 10 more
> >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> Queue
> >> -
> >> > Failed to page in more queue messages
> >> > java.lang.RuntimeException: java.lang.RuntimeException:
> >> java.io.IOException:
> >> > Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> >>
> >> > at
> >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> >> > at
> >> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> >> > at
> >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> >> > at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> >>
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> >>
> >> > at java.lang.Thread.run(Thread.java:619)
> >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
> >> locate
> >> > data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> >>
> >> > ... 9 more
> >> > Caused by: java.io.IOException: Could not locate data file
> >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> >> > at
> org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> >>
> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> >>
> >> > at
> >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> >> > at
> >> >
> >>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> >>
> >> > at
> >> >
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> >>
> >> > ... 10 more
> >> > --------------------------
> >> >
> >> > It doesn't happen consistently, but it has happened numerous times.
> The
> >> > only way to "fix" it is to kill the app, blow away the KahaDB data
> >> > directory, and restart (ugh).
> >> >
> >> > Has anybody else seen this issue? Any idea why it happens?
> >> >
> >> > Here's the embedded broker config, fwiw:
> >> > --------------------------
> >> > <broker xmlns="http://activemq.apache.org/schema/core"
> >> > id="embeddedActivemqBroker"
> >> > useJmx="false"
> >> > persistent="true"
> >> > dataDirectory="/usr/local/embedded/activemq-data"
> >> > schedulerSupport="false"
> >> > advisorySupport="false"
> >> > enableStatistics="true">
> >> > <destinationPolicy>
> >> > <policyMap>
> >> > <policyEntries>
> >> > <policyEntry queue=">" producerFlowControl="false"
> >> > memoryLimit="20mb"/>
> >> > </policyEntries>
> >> > </policyMap>
> >> > </destinationPolicy>
> >> > </broker>
> >> > --------------------------
> >> >
> >> > Help?!
> >> >
> >> > -----
> >> > Dan Checkoway
> >> > dcheckoway gmail com
> >> > --
> >> > View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
> <
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t&by-user=t>>
>
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >>
> >>
> >>
> >> --
> >> http://blog.garytully.com
> >> http://fusesource.com
> >>
> >>
> >> ------------------------------
> >> View message @
> >>
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html?by-user=t>
> >>
> >>
> >>
> >
> >
> > -----
> > Dan Checkoway
> > dcheckoway gmail com
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html?by-user=t>
>
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> http://blog.garytully.com
> http://fusesource.com
>
>
> ------------------------------
> View message @
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177622.html
>
> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>
>
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177745.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDB error: "Could not locate data file"
Posted by Gary Tully <ga...@gmail.com>.
AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems
to be different from your use case?
On 6 January 2011 15:14, dcheckoway <dc...@gmail.com> wrote:
>
> Gary,
>
> Thanks for the reply and advice. I'll definitely try disabling
> concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an IO
> exception handler.
>
> Unfortunately the issue can't reliably be reproduced. We have a cluster of
> servers all running their own local embedded queue, and it happens on one
> here, one there.
>
> That said, somebody else already opened a JIRA ticket for what looks like a
> similar/nearly identical issue:
> https://issues.apache.org/jira/browse/AMQ-3051
>
> Dan
>
>
> On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
> ml-node+3177500-224584974-140184@n4.nabble.com<ml...@n4.nabble.com>
>> wrote:
>
>> One thought, there may be an issue with the default setting of true
>> for concurrentStoreAndDispatchQueues that is causing the index to be
>> out of sync with the store.
>>
>> It is worth trying to disable that, use:
>> <persistenceAdapter>
>> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
>> concurrentStoreAndDispatchQueues="false"/>
>> </persistenceAdapter>
>>
>> How often is this occurring, to really get to the bottom of this a
>> test case will be needed.
>> Does that data file eventually get written, or are there any disk space
>> issues?
>> It may help to configure an IO exception handler just in case.
>> http://activemq.apache.org/configurable-ioexception-handling.html
>>
>> Would be a good idea to create a jira issue to track this and attach
>> what ever log and application information you have to it.
>>
>> On 6 January 2011 12:39, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
>> wrote:
>>
>> >
>> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
>> We're
>> > pumping hundreds of millions of messages per day through this thing.
>> Every
>> > once in a while, all of a sudden the KahaDB directory starts growing
>> > uncontrollably, and these errors start spewing out in the log over and
>> > over...
>> >
>> > --------------------------
>> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
>> > AbstractStoreCursor - Failed to fill batch
>> > java.lang.RuntimeException: java.io.IOException: Could not locate data
>> file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> > at
>> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> > at
>> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> > at
>> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> > at
>> >
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> > at
>> >
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> > at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.io.IOException: Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>>
>> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> > at
>> >
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> > at
>> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> > at
>> >
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> > ... 10 more
>> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue
>> -
>> > Failed to page in more queue messages
>> > java.lang.RuntimeException: java.lang.RuntimeException:
>> java.io.IOException:
>> > Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>>
>> > at
>> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
>> > at
>> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>> > at
>> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>> > at
>> >
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>
>> > at
>> >
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>> > at java.lang.Thread.run(Thread.java:619)
>> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
>> locate
>> > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>>
>> > ... 9 more
>> > Caused by: java.io.IOException: Could not locate data file
>> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>>
>> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
>> > at
>> >
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>>
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>>
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>>
>> > at
>> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>> > at
>> >
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>>
>> > at
>> >
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> > at
>> >
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>>
>> > ... 10 more
>> > --------------------------
>> >
>> > It doesn't happen consistently, but it has happened numerous times. The
>> > only way to "fix" it is to kill the app, blow away the KahaDB data
>> > directory, and restart (ugh).
>> >
>> > Has anybody else seen this issue? Any idea why it happens?
>> >
>> > Here's the embedded broker config, fwiw:
>> > --------------------------
>> > <broker xmlns="http://activemq.apache.org/schema/core"
>> > id="embeddedActivemqBroker"
>> > useJmx="false"
>> > persistent="true"
>> > dataDirectory="/usr/local/embedded/activemq-data"
>> > schedulerSupport="false"
>> > advisorySupport="false"
>> > enableStatistics="true">
>> > <destinationPolicy>
>> > <policyMap>
>> > <policyEntries>
>> > <policyEntry queue=">" producerFlowControl="false"
>> > memoryLimit="20mb"/>
>> > </policyEntries>
>> > </policyMap>
>> > </destinationPolicy>
>> > </broker>
>> > --------------------------
>> >
>> > Help?!
>> >
>> > -----
>> > Dan Checkoway
>> > dcheckoway gmail com
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>> http://fusesource.com
>>
>>
>> ------------------------------
>> View message @
>> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html
>> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>>
>>
>
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
Re: KahaDB error: "Could not locate data file"
Posted by dcheckoway <dc...@gmail.com>.
Gary,
Thanks for the reply and advice. I'll definitely try disabling
concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an IO
exception handler.
Unfortunately the issue can't reliably be reproduced. We have a cluster of
servers all running their own local embedded queue, and it happens on one
here, one there.
That said, somebody else already opened a JIRA ticket for what looks like a
similar/nearly identical issue:
https://issues.apache.org/jira/browse/AMQ-3051
Dan
On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] <
ml-node+3177500-224584974-140184@n4.nabble.com<ml...@n4.nabble.com>
> wrote:
> One thought, there may be an issue with the default setting of true
> for concurrentStoreAndDispatchQueues that is causing the index to be
> out of sync with the store.
>
> It is worth trying to disable that, use:
> <persistenceAdapter>
> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
> concurrentStoreAndDispatchQueues="false"/>
> </persistenceAdapter>
>
> How often is this occurring, to really get to the bottom of this a
> test case will be needed.
> Does that data file eventually get written, or are there any disk space
> issues?
> It may help to configure an IO exception handler just in case.
> http://activemq.apache.org/configurable-ioexception-handling.html
>
> Would be a good idea to create a jira issue to track this and attach
> what ever log and application information you have to it.
>
> On 6 January 2011 12:39, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3177500&i=0>>
> wrote:
>
> >
> > I'm using an embedded broker (version 5.4.2) with persistence enabled.
> We're
> > pumping hundreds of millions of messages per day through this thing.
> Every
> > once in a while, all of a sudden the KahaDB directory starts growing
> > uncontrollably, and these errors start spewing out in the log over and
> > over...
> >
> > --------------------------
> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> > AbstractStoreCursor - Failed to fill batch
> > java.lang.RuntimeException: java.io.IOException: Could not locate data
> file
> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> > at
> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> > at
> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> > at
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> > at
> >
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> > at
> >
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.io.IOException: Could not locate data file
> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>
> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> > at
> >
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> > at
> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> > at
> >
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> > ... 10 more
> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue
> -
> > Failed to page in more queue messages
> > java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.IOException:
> > Could not locate data file
> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
>
> > at
> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> > at
> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> > at
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> > at
> >
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>
> > at
> >
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not
> locate
> > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
>
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
>
> > ... 9 more
> > Caused by: java.io.IOException: Could not locate data file
> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
>
> > at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> > at
> >
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
>
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
>
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
>
> > at
> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> > at
> >
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
>
> > at
> >
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> > at
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
>
> > ... 10 more
> > --------------------------
> >
> > It doesn't happen consistently, but it has happened numerous times. The
> > only way to "fix" it is to kill the app, blow away the KahaDB data
> > directory, and restart (ugh).
> >
> > Has anybody else seen this issue? Any idea why it happens?
> >
> > Here's the embedded broker config, fwiw:
> > --------------------------
> > <broker xmlns="http://activemq.apache.org/schema/core"
> > id="embeddedActivemqBroker"
> > useJmx="false"
> > persistent="true"
> > dataDirectory="/usr/local/embedded/activemq-data"
> > schedulerSupport="false"
> > advisorySupport="false"
> > enableStatistics="true">
> > <destinationPolicy>
> > <policyMap>
> > <policyEntries>
> > <policyEntry queue=">" producerFlowControl="false"
> > memoryLimit="20mb"/>
> > </policyEntries>
> > </policyMap>
> > </destinationPolicy>
> > </broker>
> > --------------------------
> >
> > Help?!
> >
> > -----
> > Dan Checkoway
> > dcheckoway gmail com
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t>
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> http://blog.garytully.com
> http://fusesource.com
>
>
> ------------------------------
> View message @
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html
> To unsubscribe from KahaDB error: "Could not locate data file", click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>.
>
>
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: KahaDB error: "Could not locate data file"
Posted by Gary Tully <ga...@gmail.com>.
One thought, there may be an issue with the default setting of true
for concurrentStoreAndDispatchQueues that is causing the index to be
out of sync with the store.
It is worth trying to disable that, use:
<persistenceAdapter>
<kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
concurrentStoreAndDispatchQueues="false"/>
</persistenceAdapter>
How often is this occurring, to really get to the bottom of this a
test case will be needed.
Does that data file eventually get written, or are there any disk space issues?
It may help to configure an IO exception handler just in case.
http://activemq.apache.org/configurable-ioexception-handling.html
Would be a good idea to create a jira issue to track this and attach
what ever log and application information you have to it.
On 6 January 2011 12:39, dcheckoway <dc...@gmail.com> wrote:
>
> I'm using an embedded broker (version 5.4.2) with persistence enabled. We're
> pumping hundreds of millions of messages per day through this thing. Every
> once in a while, all of a sudden the KahaDB directory starts growing
> uncontrollably, and these errors start spewing out in the log over and
> over...
>
> --------------------------
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> AbstractStoreCursor - Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
> Failed to page in more queue messages
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
> Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
> data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> ... 9 more
> Caused by: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> --------------------------
>
> It doesn't happen consistently, but it has happened numerous times. The
> only way to "fix" it is to kill the app, blow away the KahaDB data
> directory, and restart (ugh).
>
> Has anybody else seen this issue? Any idea why it happens?
>
> Here's the embedded broker config, fwiw:
> --------------------------
> <broker xmlns="http://activemq.apache.org/schema/core"
> id="embeddedActivemqBroker"
> useJmx="false"
> persistent="true"
> dataDirectory="/usr/local/embedded/activemq-data"
> schedulerSupport="false"
> advisorySupport="false"
> enableStatistics="true">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="20mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> </broker>
> --------------------------
>
> Help?!
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com