You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "TRULIS Timothy (Tim)" <ti...@hexagon.com> on 2021/11/23 13:09:53 UTC
KahaDB Corruption
Hello all,
I am running KahaDB v6 w/ ActiveMQ v5.15.12 in Docker container with mapped data/conf volumes to host.
Starting the container, I receive the following errors.
ERROR | [0:ActiveMQ.DLQ] references corrupt locations
ERROR | Failed to start Apache ActiveMQ (localhost, null)
java.io.IOException: Detected missing/corrupt journal files referenced by:[0:ActiveMQ.DLQ] 203 messages affected.
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:988)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:726)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:787)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:723)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:482)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:502)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:306)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:221)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.12.jar:5.15.12]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:693)[activemq-broker-5.15.12.jar:5.15.12]
at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:677)[activemq-broker-5.15.12.jar:5.15.12]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:641)[activemq-broker-5.15.12.jar:5.15.12]
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.12.jar:5.15.12]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566)[:]
I did search the user archives but couldn't find anything helpful.
I did find this: https://issues.apache.org/jira/browse/AMQ-6831 and noted the workaround in the thread of disabling the checkForCorruptJournalFiles to avoid the problem as well as increasing the journalMaxWriteBatchSize parameter.
I am going to try these workarounds but am wondering if there is any other ways to troubleshoot KahaDB to fix this issue?
Thank you,
Tim
Re: KahaDB Corruption
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you decide to try rebuilding the KahaDB index without the corrupted
file(s), https://access.redhat.com/solutions/276323 gives steps for doing
that. I'd highly recommend you back up the KahaDB data directory and that
you try the process in a development broker using a copy of the data files.
Note that if a data file really is corrupt, you're almost certainly going
to lose messages. If you're willing to lose all messages in order to get
back up and running fast, you can just delete all the KahaDB files and
start from an empty message store. If you're not willing to do that, then
I'd try to figure out which files are corrupt (the Log4j config at
http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
could help here) and delete them along with the index and hope that gets
KahaDB to a point where it can recover with minimal message loss.
Tim
On Tue, Nov 23, 2021, 6:12 AM JB Onofré <jb...@nanthrax.net> wrote:
> Hi Tim
>
> Maybe you have a process that remove or compress kahadb journal log files
> (something like logrotate for instance ) ?
>
> Regards
> JB
>
> > Le 23 nov. 2021 à 14:10, TRULIS Timothy (Tim) <
> timothy.trulis@hexagon.com> a écrit :
> >
> > Hello all,
> >
> > I am running KahaDB v6 w/ ActiveMQ v5.15.12 in Docker container with
> mapped data/conf volumes to host.
> >
> > Starting the container, I receive the following errors.
> >
> > ERROR | [0:ActiveMQ.DLQ] references corrupt locations
> > ERROR | Failed to start Apache ActiveMQ (localhost, null)
> > java.io.IOException: Detected missing/corrupt journal files referenced
> by:[0:ActiveMQ.DLQ] 203 messages affected.
> > at
> org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:988)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:726)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:787)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:723)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:482)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:502)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:306)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:221)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:693)[activemq-broker-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:677)[activemq-broker-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:641)[activemq-broker-5.15.12.jar:5.15.12]
> > at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.12.jar:5.15.12]
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:]
> > at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:]
> > at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:]
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)[:]
> >
> > I did search the user archives but couldn't find anything helpful.
> >
> > I did find this: https://issues.apache.org/jira/browse/AMQ-6831 and
> noted the workaround in the thread of disabling the
> checkForCorruptJournalFiles to avoid the problem as well as increasing the
> journalMaxWriteBatchSize parameter.
> >
> > I am going to try these workarounds but am wondering if there is any
> other ways to troubleshoot KahaDB to fix this issue?
> >
> >
> > Thank you,
> > Tim
>
>
Re: KahaDB Corruption
Posted by JB Onofré <jb...@nanthrax.net>.
Hi Tim
Maybe you have a process that remove or compress kahadb journal log files (something like logrotate for instance ) ?
Regards
JB
> Le 23 nov. 2021 à 14:10, TRULIS Timothy (Tim) <ti...@hexagon.com> a écrit :
>
> Hello all,
>
> I am running KahaDB v6 w/ ActiveMQ v5.15.12 in Docker container with mapped data/conf volumes to host.
>
> Starting the container, I receive the following errors.
>
> ERROR | [0:ActiveMQ.DLQ] references corrupt locations
> ERROR | Failed to start Apache ActiveMQ (localhost, null)
> java.io.IOException: Detected missing/corrupt journal files referenced by:[0:ActiveMQ.DLQ] 203 messages affected.
> at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:988)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:726)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:787)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:723)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:482)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:502)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:306)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:221)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.12.jar:5.15.12]
> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.12.jar:5.15.12]
> at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:693)[activemq-broker-5.15.12.jar:5.15.12]
> at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:677)[activemq-broker-5.15.12.jar:5.15.12]
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:641)[activemq-broker-5.15.12.jar:5.15.12]
> at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.12.jar:5.15.12]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:]
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)[:]
>
> I did search the user archives but couldn't find anything helpful.
>
> I did find this: https://issues.apache.org/jira/browse/AMQ-6831 and noted the workaround in the thread of disabling the checkForCorruptJournalFiles to avoid the problem as well as increasing the journalMaxWriteBatchSize parameter.
>
> I am going to try these workarounds but am wondering if there is any other ways to troubleshoot KahaDB to fix this issue?
>
>
> Thank you,
> Tim