You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Oleg Dulin <ol...@gmail.com> on 2013/11/03 14:25:04 UTC
AMQ 5.8.1 failed to restart after a hardware crash
I believe my issue is related to this:
https://issues.apache.org/jira/browse/AMQ-3422
I got this exception:
java.lang.RuntimeException: java.io.IOException: Invalid location:
8249:28018848, : java.lang.NegativeArraySizeException
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1775)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2003)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1491)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Invalid location: 8249:28018848, :
java.lang.NegativeArraySizeException
at
org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
at
org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1029)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:557)
at
org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:546)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
... 10 more
I couldn't restart no matter what. It's silly that kahadb got corrupted
because of a crash, defeats the whole purpose of a disk journal.
I ended up clearing out data directory and restarting, but that is suboptimal.
Any thoughts ?
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: AMQ 5.8.1 failed to restart after a hardware crash
Posted by Christian Posta <ch...@gmail.com>.
What are your settings on kahadb persistence adapter? You can add
'checksumJournalFiles' and 'checkForCorruptJournalFiles' to skip
corrupt sections (however they may have been corrupted?). there is a
write-ahead log which should remedy failed or in-flight transactions.
On Sun, Nov 3, 2013 at 6:25 AM, Oleg Dulin <ol...@gmail.com> wrote:
> I believe my issue is related to this:
>
> https://issues.apache.org/jira/browse/AMQ-3422
>
> I got this exception:
>
> java.lang.RuntimeException: java.io.IOException: Invalid location:
> 8249:28018848, : java.lang.NegativeArraySizeException
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
> at
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1775)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2003)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1491)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Invalid location: 8249:28018848, :
> java.lang.NegativeArraySizeException
> at
> org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
> at
> org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1029)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:557)
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:546)
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
> ... 10 more
>
>
> I couldn't restart no matter what. It's silly that kahadb got corrupted
> because of a crash, defeats the whole purpose of a disk journal.
>
> I ended up clearing out data directory and restarting, but that is
> suboptimal.
>
> Any thoughts ?
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Re: AMQ 5.8.0 failed to restart after a hardware crash
Posted by Oleg Dulin <ol...@gmail.com>.
It's 5.8.0, my bad.
Unfortunately, upgrading is not that easy since this is a production system.
So, any ideas other than that ? How do I know this bug was fixed (
https://issues.apache.org/jira/browse/AMQ-3422 ) ? JIRA says it's been
fixed, but obviously in 5.8.0 it's not.
Oleg
On 2013-11-03 15:00:43 +0000, Timothy Bish said:
> On 11/03/2013 08:25 AM, Oleg Dulin wrote:
>> I believe my issue is related to this:
>>
>> https://issues.apache.org/jira/browse/AMQ-3422
>>
>> I got this exception:
>>
>> java.lang.RuntimeException: java.io.IOException: Invalid location:
>> 8249:28018848, : java.lang.NegativeArraySizeException
>> at
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
>>
>> at
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
>>
>> at
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
>>
>> at
>> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1775)
>>
>> at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2003)
>> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1491)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
>>
>> at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.io.IOException: Invalid location: 8249:28018848, :
>> java.lang.NegativeArraySizeException
>> at
>> org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
>>
>> at
>> org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604)
>>
>> at
>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961)
>>
>> at
>> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1029)
>>
>> at
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:557)
>>
>> at
>> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
>>
>> at
>> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:546)
>>
>> at
>> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
>>
>> at
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>>
>> at
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
>>
>> ... 10 more
>>
>>
>> I couldn't restart no matter what. It's silly that kahadb got corrupted
>> because of a crash, defeats the whole purpose of a disk journal.
>>
>> I ended up clearing out data directory and restarting, but that is suboptimal.
>>
>> Any thoughts ?
>>
>>
> Update your version to v5.9.0, no idea where you would get a v5.8.1
> since there isn't one.
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: AMQ 5.8.1 failed to restart after a hardware crash
Posted by Timothy Bish <ta...@gmail.com>.
On 11/03/2013 08:25 AM, Oleg Dulin wrote:
> I believe my issue is related to this:
>
> https://issues.apache.org/jira/browse/AMQ-3422
>
> I got this exception:
>
> java.lang.RuntimeException: java.io.IOException: Invalid location:
> 8249:28018848, : java.lang.NegativeArraySizeException
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
>
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
>
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
>
> at
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1775)
>
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2003)
> at
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1491)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
>
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Invalid location: 8249:28018848, :
> java.lang.NegativeArraySizeException
> at
> org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
>
> at
> org.apache.activemq.store.kahadb.disk.journal.Journal.read(Journal.java:604)
>
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:961)
>
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1029)
>
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:557)
>
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
>
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:546)
>
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
>
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
>
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
>
> ... 10 more
>
>
> I couldn't restart no matter what. It's silly that kahadb got
> corrupted because of a crash, defeats the whole purpose of a disk
> journal.
>
> I ended up clearing out data directory and restarting, but that is
> suboptimal.
>
> Any thoughts ?
>
>
Update your version to v5.9.0, no idea where you would get a v5.8.1
since there isn't one.
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/