You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jonathan Roby <Jo...@umanitoba.ca> on 2019/11/02 17:14:05 UTC
org.apache.activemq.kaha.impl.index.?
Hi, i've just joined the list so apologies if i've not this correctly.
We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
Thank you for any help anyone can give me.
Jon
Here is the last part of the stack trace.
Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
hi, sorry for the delay
its a large storage array mounted onto the VM the software runs upon
Jon
________________________________
From: Jean-Baptiste Onofré <jb...@nanthrax.net>
Sent: Saturday, November 2, 2019 12:46:22 PM
To: users@activemq.apache.org
Subject: Re: org.apache.activemq.kaha.impl.index.?
Hi,
It's definitely something weird while reading file on the filesystem.
Is it a local filesystem or shared one (NFS, CIFS, ...) ?
Regards
JB
On 02/11/2019 18:18, Jonathan Roby wrote:
> Hi
>
>
> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Saturday, November 2, 2019 12:16:31 PM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi,
>
> Does the kahadb folder has been created with the same ActiveMQ version
> (and you updated ActiveMQ version later) ?
>
> It seems the kahadb index is not consistent.
>
> Regards
> JB
>
> On 02/11/2019 18:14, Jonathan Roby wrote:
>> Hi, i've just joined the list so apologies if i've not this correctly.
>>
>>
>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>
>>
>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>
>>
>> Thank you for any help anyone can give me.
>>
>> Jon
>>
>>
>> Here is the last part of the stack trace.
>>
>>
>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks for the update !
Great you figured it out and always happy to help.
Regards
JB
On 05/11/2019 16:27, Jonathan Roby wrote:
> Hi All
>
>
> just an update: I figured out the issue. It was due to the software we use (fedora-commons/tomcat) using its own set of configuration files and data directories. So I was trying to debug the wrong set of config files and data directories, thats why nothing seemed to work. And in the end, it was just deleting the data directory.
>
>
> Thanks Jean-Baptiste and Tim for your help
>
> Jon
>
> ________________________________
> From: Tim Bain <tb...@alumni.duke.edu>
> Sent: Tuesday, November 5, 2019 7:34:10 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Is there any chance that a JAR from a newer version of ActiveMQ ended up on
> the broker's classpath? If you delete the contents of the lib directory and
> replace it with the contents from a freshly installed copy of 5.7.0, does
> the behavior change?
>
> Also, can you share your activemq.xml with us? And does this behavior
> change if you use the copy of activemq.xml from a fresh install?
>
> Tim
>
> On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <Jo...@umanitoba.ca>
> wrote:
>
>> Hi
>>
>>
>> I tried putting the kahadb directory on the local directory, it
>> regenerated on startup but still gave me a BadMagicException
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Sent: Sunday, November 3, 2019 11:07:22 AM
>> To: users@activemq.apache.org
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi Jonathan,
>>
>> user mailing list is fine for this kind of question.
>>
>> IMHO, I'm pretty sure it's related to the filesystem.
>>
>> As Tim suggested, maybe you can try to put the kahadb directory on local
>> filesystem instead of the SAN. It could be an interesting test.
>>
>> Regards
>> JB
>>
>> On 03/11/2019 16:52, Jonathan Roby wrote:
>>> Hi
>>>
>>>
>>> Also, i'm sorry for the subject heading of my original email, I'm not
>> sure what happened but its somewhat nonsensical.
>>>
>>>
>>> Also, would anyone recommend I post this question to the developer
>> mailing list, in case they might have other ideas?
>>>
>>>
>>> I've asked our IT infrastructure team if they can check the VMs
>> storage/machine layout for recent changes that could have caused this.
>> hopefully t hey find something or I get a lucky break
>>>
>>>
>>> Thank you everyone for your help so far
>>>
>>> Jon
>>>
>>> ________________________________
>>> From: Jonathan Roby
>>> Sent: Sunday, November 3, 2019 9:02:56 AM
>>> To: ActiveMQ Users
>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>
>>>
>>> Hi
>>>
>>>
>>> I just checked and the directory is created exactly when the process
>> tries to start. I've also checked and the ports used by the broker are not
>> visible in the output of the netstat command.
>>>
>>>
>>> I also tried moving the kahadb directory onto the local drive and still
>> get the BadMagicException at the end of nested exception stacktrace
>>>
>>>
>>> Thank you
>>>
>>> Jon
>>>
>>> ________________________________
>>> From: Tim Bain <tb...@alumni.duke.edu>
>>> Sent: Sunday, November 3, 2019 8:06:29 AM
>>> To: ActiveMQ Users
>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>
>>> Following up on JB's line of questioning, is the directory only recreated
>>> when (exactly when, to the second, with a matching timestamp) this broker
>>> process is started? Might it be possible that a different broker process,
>>> running elsewhere, might be creating the directory and its files?
>>>
>>> If you reconfigure this broker to store the KahaDB data file on local
>> disk
>>> instead of your SAN (just temporarily), does the process start
>>> successfully?
>>>
>>> Tim
>>>
>>> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
>>> wrote:
>>>
>>>>
>>>> No, its not fixed. I wiped out the data folder and the persistant
>> folder.
>>>> both were regenerated but still is failing
>>>>
>>>>
>>>> its weird as we have other instances set up exactly the same as this one
>>>> that dont have this error. same versions, same everything
>>>>
>>>>
>>>> so it sounds like you might be correct about it being the filesystem. i
>>>> hope not as that can be tricky to debug
>>>>
>>>>
>>>> Thank you for your help
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> Sent: Saturday, November 2, 2019 12:48:29 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> I guess it's fixed with you complete remove the kahadb folder right ?
>>>>
>>>> I remember couple of users mentioned similar issue on virtual machine.
>>>>
>>>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>>>> on this version around kahadb.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>>>> Hi,
>>>>>
>>>>> It's definitely something weird while reading file on the filesystem.
>>>>>
>>>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>>>> now deleted the kahadb directory a few times (it regenerated) and even
>>>> moved the broker path in case it was a drive error
>>>>>>
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> ________________________________
>>>>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>>>> To: users@activemq.apache.org
>>>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>>>> (and you updated ActiveMQ version later) ?
>>>>>>
>>>>>> It seems the kahadb index is not consistent.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>>>> Hi, i've just joined the list so apologies if i've not this
>> correctly.
>>>>>>>
>>>>>>>
>>>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>>>> message passing between components. Yesterday, one of our instances
>> seemed
>>>> to stop working. after several restarts of components and the system
>>>> itself, I noticed that when the components restarted, it always
>> produced a
>>>> nested exception and the very last error was
>>>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>>>
>>>>>>>
>>>>>>> I've googled this and taken the advice given, which is usually to
>>>> empty/delete the data directory or the kahadb directory and it still
>> isnt
>>>> working. All the other components seem to be working fine.
>>>>>>>
>>>>>>>
>>>>>>> Thank you for any help anyone can give me.
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>>
>>>>>>> Here is the last part of the stack trace.
>>>>>>>
>>>>>>>
>>>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
>> null
>>>>>>> at
>>>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at
>>>>
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> ~[na:1.8.0_162]
>>>>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> ~[na:1.8.0_162]
>>>>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> ~[na:1.8.0_162]
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>> ~[na:1.8.0_162]
>>>>>>> at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>> at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>> at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>> Talend - A Cloud Data Integration Leader (modern ETL)<
>>>> http://www.talend.com/>
>>>> www.talend.com<http://www.talend.com>
>>>> Talend Data Fabric offers a single suite of cloud apps for data
>>>> integration and data integrity to help enterprises collect, govern,
>>>> transform, and share data.
>>>>
>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>> Talend - A Cloud Data Integration Leader (modern ETL)<
>>>> http://www.talend.com/>
>>>> www.talend.com<http://www.talend.com>
>>>> Talend Data Fabric offers a single suite of cloud apps for data
>>>> integration and data integrity to help enterprises collect, govern,
>>>> transform, and share data.
>>>>
>>>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Tim Bain <tb...@alumni.duke.edu>.
Great, I'm glad you got it sorted out. Thanks for closing the loop and
letting us know.
Tim
On Tue, Nov 5, 2019, 8:27 AM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
> Hi All
>
>
> just an update: I figured out the issue. It was due to the software we
> use (fedora-commons/tomcat) using its own set of configuration files and
> data directories. So I was trying to debug the wrong set of config files
> and data directories, thats why nothing seemed to work. And in the end, it
> was just deleting the data directory.
>
>
> Thanks Jean-Baptiste and Tim for your help
>
> Jon
>
> ________________________________
> From: Tim Bain <tb...@alumni.duke.edu>
> Sent: Tuesday, November 5, 2019 7:34:10 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Is there any chance that a JAR from a newer version of ActiveMQ ended up on
> the broker's classpath? If you delete the contents of the lib directory and
> replace it with the contents from a freshly installed copy of 5.7.0, does
> the behavior change?
>
> Also, can you share your activemq.xml with us? And does this behavior
> change if you use the copy of activemq.xml from a fresh install?
>
> Tim
>
> On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <Jo...@umanitoba.ca>
> wrote:
>
> > Hi
> >
> >
> > I tried putting the kahadb directory on the local directory, it
> > regenerated on startup but still gave me a BadMagicException
> >
> > ________________________________
> > From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > Sent: Sunday, November 3, 2019 11:07:22 AM
> > To: users@activemq.apache.org
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Hi Jonathan,
> >
> > user mailing list is fine for this kind of question.
> >
> > IMHO, I'm pretty sure it's related to the filesystem.
> >
> > As Tim suggested, maybe you can try to put the kahadb directory on local
> > filesystem instead of the SAN. It could be an interesting test.
> >
> > Regards
> > JB
> >
> > On 03/11/2019 16:52, Jonathan Roby wrote:
> > > Hi
> > >
> > >
> > > Also, i'm sorry for the subject heading of my original email, I'm not
> > sure what happened but its somewhat nonsensical.
> > >
> > >
> > > Also, would anyone recommend I post this question to the developer
> > mailing list, in case they might have other ideas?
> > >
> > >
> > > I've asked our IT infrastructure team if they can check the VMs
> > storage/machine layout for recent changes that could have caused this.
> > hopefully t hey find something or I get a lucky break
> > >
> > >
> > > Thank you everyone for your help so far
> > >
> > > Jon
> > >
> > > ________________________________
> > > From: Jonathan Roby
> > > Sent: Sunday, November 3, 2019 9:02:56 AM
> > > To: ActiveMQ Users
> > > Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >
> > >
> > > Hi
> > >
> > >
> > > I just checked and the directory is created exactly when the process
> > tries to start. I've also checked and the ports used by the broker are
> not
> > visible in the output of the netstat command.
> > >
> > >
> > > I also tried moving the kahadb directory onto the local drive and still
> > get the BadMagicException at the end of nested exception stacktrace
> > >
> > >
> > > Thank you
> > >
> > > Jon
> > >
> > > ________________________________
> > > From: Tim Bain <tb...@alumni.duke.edu>
> > > Sent: Sunday, November 3, 2019 8:06:29 AM
> > > To: ActiveMQ Users
> > > Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >
> > > Following up on JB's line of questioning, is the directory only
> recreated
> > > when (exactly when, to the second, with a matching timestamp) this
> broker
> > > process is started? Might it be possible that a different broker
> process,
> > > running elsewhere, might be creating the directory and its files?
> > >
> > > If you reconfigure this broker to store the KahaDB data file on local
> > disk
> > > instead of your SAN (just temporarily), does the process start
> > > successfully?
> > >
> > > Tim
> > >
> > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jonathan.Roby@umanitoba.ca
> >
> > > wrote:
> > >
> > >>
> > >> No, its not fixed. I wiped out the data folder and the persistant
> > folder.
> > >> both were regenerated but still is failing
> > >>
> > >>
> > >> its weird as we have other instances set up exactly the same as this
> one
> > >> that dont have this error. same versions, same everything
> > >>
> > >>
> > >> so it sounds like you might be correct about it being the filesystem.
> i
> > >> hope not as that can be tricky to debug
> > >>
> > >>
> > >> Thank you for your help
> > >>
> > >> Jon
> > >>
> > >> ________________________________
> > >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >> Sent: Saturday, November 2, 2019 12:48:29 PM
> > >> To: users@activemq.apache.org
> > >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >>
> > >> I guess it's fixed with you complete remove the kahadb folder right ?
> > >>
> > >> I remember couple of users mentioned similar issue on virtual machine.
> > >>
> > >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and
> improvements
> > >> on this version around kahadb.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > >>> Hi,
> > >>>
> > >>> It's definitely something weird while reading file on the filesystem.
> > >>>
> > >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> > >>>
> > >>> Regards
> > >>> JB
> > >>>
> > >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> > >>>> Hi
> > >>>>
> > >>>>
> > >>>> as far as I know, the version of ActiveMQ hasn't been updated and
> i've
> > >> now deleted the kahadb directory a few times (it regenerated) and even
> > >> moved the broker path in case it was a drive error
> > >>>>
> > >>>>
> > >>>> Jon
> > >>>>
> > >>>> ________________________________
> > >>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> > >>>> To: users@activemq.apache.org
> > >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> Does the kahadb folder has been created with the same ActiveMQ
> version
> > >>>> (and you updated ActiveMQ version later) ?
> > >>>>
> > >>>> It seems the kahadb index is not consistent.
> > >>>>
> > >>>> Regards
> > >>>> JB
> > >>>>
> > >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> > >>>>> Hi, i've just joined the list so apologies if i've not this
> > correctly.
> > >>>>>
> > >>>>>
> > >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> > >> message passing between components. Yesterday, one of our instances
> > seemed
> > >> to stop working. after several restarts of components and the system
> > >> itself, I noticed that when the components restarted, it always
> > produced a
> > >> nested exception and the very last error was
> > >> org.apache.activemq.kaha.impl.index.BadMagicException.
> > >>>>>
> > >>>>>
> > >>>>> I've googled this and taken the advice given, which is usually to
> > >> empty/delete the data directory or the kahadb directory and it still
> > isnt
> > >> working. All the other components seem to be working fine.
> > >>>>>
> > >>>>>
> > >>>>> Thank you for any help anyone can give me.
> > >>>>>
> > >>>>> Jon
> > >>>>>
> > >>>>>
> > >>>>> Here is the last part of the stack trace.
> > >>>>>
> > >>>>>
> > >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> > null
> > >>>>> at
> > >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> > org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at
> > >>
> >
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> ~[na:1.8.0_162]
> > >>>>> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >> ~[na:1.8.0_162]
> > >>>>> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> ~[na:1.8.0_162]
> > >>>>> at java.lang.reflect.Method.invoke(Method.java:498)
> > ~[na:1.8.0_162]
> > >>>>> at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>> at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>> at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Jean-Baptiste Onofré
> > >>>> jbonofre@apache.org
> > >>>> http://blog.nanthrax.net
> > >>>> Talend - http://www.talend.com
> > >> Talend - A Cloud Data Integration Leader (modern ETL)<
> > >> http://www.talend.com/>
> > >> www.talend.com<http://www.talend.com>
> > >> Talend Data Fabric offers a single suite of cloud apps for data
> > >> integration and data integrity to help enterprises collect, govern,
> > >> transform, and share data.
> > >>
> > >>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbonofre@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >> Talend - A Cloud Data Integration Leader (modern ETL)<
> > >> http://www.talend.com/>
> > >> www.talend.com<http://www.talend.com>
> > >> Talend Data Fabric offers a single suite of cloud apps for data
> > >> integration and data integrity to help enterprises collect, govern,
> > >> transform, and share data.
> > >>
> > >>
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
Hi All
just an update: I figured out the issue. It was due to the software we use (fedora-commons/tomcat) using its own set of configuration files and data directories. So I was trying to debug the wrong set of config files and data directories, thats why nothing seemed to work. And in the end, it was just deleting the data directory.
Thanks Jean-Baptiste and Tim for your help
Jon
________________________________
From: Tim Bain <tb...@alumni.duke.edu>
Sent: Tuesday, November 5, 2019 7:34:10 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?
Is there any chance that a JAR from a newer version of ActiveMQ ended up on
the broker's classpath? If you delete the contents of the lib directory and
replace it with the contents from a freshly installed copy of 5.7.0, does
the behavior change?
Also, can you share your activemq.xml with us? And does this behavior
change if you use the copy of activemq.xml from a fresh install?
Tim
On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
> Hi
>
>
> I tried putting the kahadb directory on the local directory, it
> regenerated on startup but still gave me a BadMagicException
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Sunday, November 3, 2019 11:07:22 AM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi Jonathan,
>
> user mailing list is fine for this kind of question.
>
> IMHO, I'm pretty sure it's related to the filesystem.
>
> As Tim suggested, maybe you can try to put the kahadb directory on local
> filesystem instead of the SAN. It could be an interesting test.
>
> Regards
> JB
>
> On 03/11/2019 16:52, Jonathan Roby wrote:
> > Hi
> >
> >
> > Also, i'm sorry for the subject heading of my original email, I'm not
> sure what happened but its somewhat nonsensical.
> >
> >
> > Also, would anyone recommend I post this question to the developer
> mailing list, in case they might have other ideas?
> >
> >
> > I've asked our IT infrastructure team if they can check the VMs
> storage/machine layout for recent changes that could have caused this.
> hopefully t hey find something or I get a lucky break
> >
> >
> > Thank you everyone for your help so far
> >
> > Jon
> >
> > ________________________________
> > From: Jonathan Roby
> > Sent: Sunday, November 3, 2019 9:02:56 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> >
> > Hi
> >
> >
> > I just checked and the directory is created exactly when the process
> tries to start. I've also checked and the ports used by the broker are not
> visible in the output of the netstat command.
> >
> >
> > I also tried moving the kahadb directory onto the local drive and still
> get the BadMagicException at the end of nested exception stacktrace
> >
> >
> > Thank you
> >
> > Jon
> >
> > ________________________________
> > From: Tim Bain <tb...@alumni.duke.edu>
> > Sent: Sunday, November 3, 2019 8:06:29 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Following up on JB's line of questioning, is the directory only recreated
> > when (exactly when, to the second, with a matching timestamp) this broker
> > process is started? Might it be possible that a different broker process,
> > running elsewhere, might be creating the directory and its files?
> >
> > If you reconfigure this broker to store the KahaDB data file on local
> disk
> > instead of your SAN (just temporarily), does the process start
> > successfully?
> >
> > Tim
> >
> > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
> > wrote:
> >
> >>
> >> No, its not fixed. I wiped out the data folder and the persistant
> folder.
> >> both were regenerated but still is failing
> >>
> >>
> >> its weird as we have other instances set up exactly the same as this one
> >> that dont have this error. same versions, same everything
> >>
> >>
> >> so it sounds like you might be correct about it being the filesystem. i
> >> hope not as that can be tricky to debug
> >>
> >>
> >> Thank you for your help
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Sent: Saturday, November 2, 2019 12:48:29 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> I guess it's fixed with you complete remove the kahadb folder right ?
> >>
> >> I remember couple of users mentioned similar issue on virtual machine.
> >>
> >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> >> on this version around kahadb.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> >>> Hi,
> >>>
> >>> It's definitely something weird while reading file on the filesystem.
> >>>
> >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> >>>> Hi
> >>>>
> >>>>
> >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
> >> now deleted the kahadb directory a few times (it regenerated) and even
> >> moved the broker path in case it was a drive error
> >>>>
> >>>>
> >>>> Jon
> >>>>
> >>>> ________________________________
> >>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> >>>> To: users@activemq.apache.org
> >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>>>
> >>>> Hi,
> >>>>
> >>>> Does the kahadb folder has been created with the same ActiveMQ version
> >>>> (and you updated ActiveMQ version later) ?
> >>>>
> >>>> It seems the kahadb index is not consistent.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>>>> Hi, i've just joined the list so apologies if i've not this
> correctly.
> >>>>>
> >>>>>
> >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> >> message passing between components. Yesterday, one of our instances
> seemed
> >> to stop working. after several restarts of components and the system
> >> itself, I noticed that when the components restarted, it always
> produced a
> >> nested exception and the very last error was
> >> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>>>
> >>>>>
> >>>>> I've googled this and taken the advice given, which is usually to
> >> empty/delete the data directory or the kahadb directory and it still
> isnt
> >> working. All the other components seem to be working fine.
> >>>>>
> >>>>>
> >>>>> Thank you for any help anyone can give me.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>>
> >>>>> Here is the last part of the stack trace.
> >>>>>
> >>>>>
> >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> null
> >>>>> at
> >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> ~[na:1.8.0_162]
> >>>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> ~[na:1.8.0_162]
> >>>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> ~[na:1.8.0_162]
> >>>>> at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_162]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>>>
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Tim Bain <tb...@alumni.duke.edu>.
Is there any chance that a JAR from a newer version of ActiveMQ ended up on
the broker's classpath? If you delete the contents of the lib directory and
replace it with the contents from a freshly installed copy of 5.7.0, does
the behavior change?
Also, can you share your activemq.xml with us? And does this behavior
change if you use the copy of activemq.xml from a fresh install?
Tim
On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
> Hi
>
>
> I tried putting the kahadb directory on the local directory, it
> regenerated on startup but still gave me a BadMagicException
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Sunday, November 3, 2019 11:07:22 AM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi Jonathan,
>
> user mailing list is fine for this kind of question.
>
> IMHO, I'm pretty sure it's related to the filesystem.
>
> As Tim suggested, maybe you can try to put the kahadb directory on local
> filesystem instead of the SAN. It could be an interesting test.
>
> Regards
> JB
>
> On 03/11/2019 16:52, Jonathan Roby wrote:
> > Hi
> >
> >
> > Also, i'm sorry for the subject heading of my original email, I'm not
> sure what happened but its somewhat nonsensical.
> >
> >
> > Also, would anyone recommend I post this question to the developer
> mailing list, in case they might have other ideas?
> >
> >
> > I've asked our IT infrastructure team if they can check the VMs
> storage/machine layout for recent changes that could have caused this.
> hopefully t hey find something or I get a lucky break
> >
> >
> > Thank you everyone for your help so far
> >
> > Jon
> >
> > ________________________________
> > From: Jonathan Roby
> > Sent: Sunday, November 3, 2019 9:02:56 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> >
> > Hi
> >
> >
> > I just checked and the directory is created exactly when the process
> tries to start. I've also checked and the ports used by the broker are not
> visible in the output of the netstat command.
> >
> >
> > I also tried moving the kahadb directory onto the local drive and still
> get the BadMagicException at the end of nested exception stacktrace
> >
> >
> > Thank you
> >
> > Jon
> >
> > ________________________________
> > From: Tim Bain <tb...@alumni.duke.edu>
> > Sent: Sunday, November 3, 2019 8:06:29 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Following up on JB's line of questioning, is the directory only recreated
> > when (exactly when, to the second, with a matching timestamp) this broker
> > process is started? Might it be possible that a different broker process,
> > running elsewhere, might be creating the directory and its files?
> >
> > If you reconfigure this broker to store the KahaDB data file on local
> disk
> > instead of your SAN (just temporarily), does the process start
> > successfully?
> >
> > Tim
> >
> > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
> > wrote:
> >
> >>
> >> No, its not fixed. I wiped out the data folder and the persistant
> folder.
> >> both were regenerated but still is failing
> >>
> >>
> >> its weird as we have other instances set up exactly the same as this one
> >> that dont have this error. same versions, same everything
> >>
> >>
> >> so it sounds like you might be correct about it being the filesystem. i
> >> hope not as that can be tricky to debug
> >>
> >>
> >> Thank you for your help
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Sent: Saturday, November 2, 2019 12:48:29 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> I guess it's fixed with you complete remove the kahadb folder right ?
> >>
> >> I remember couple of users mentioned similar issue on virtual machine.
> >>
> >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> >> on this version around kahadb.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> >>> Hi,
> >>>
> >>> It's definitely something weird while reading file on the filesystem.
> >>>
> >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> >>>> Hi
> >>>>
> >>>>
> >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
> >> now deleted the kahadb directory a few times (it regenerated) and even
> >> moved the broker path in case it was a drive error
> >>>>
> >>>>
> >>>> Jon
> >>>>
> >>>> ________________________________
> >>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> >>>> To: users@activemq.apache.org
> >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>>>
> >>>> Hi,
> >>>>
> >>>> Does the kahadb folder has been created with the same ActiveMQ version
> >>>> (and you updated ActiveMQ version later) ?
> >>>>
> >>>> It seems the kahadb index is not consistent.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>>>> Hi, i've just joined the list so apologies if i've not this
> correctly.
> >>>>>
> >>>>>
> >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> >> message passing between components. Yesterday, one of our instances
> seemed
> >> to stop working. after several restarts of components and the system
> >> itself, I noticed that when the components restarted, it always
> produced a
> >> nested exception and the very last error was
> >> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>>>
> >>>>>
> >>>>> I've googled this and taken the advice given, which is usually to
> >> empty/delete the data directory or the kahadb directory and it still
> isnt
> >> working. All the other components seem to be working fine.
> >>>>>
> >>>>>
> >>>>> Thank you for any help anyone can give me.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>>
> >>>>> Here is the last part of the stack trace.
> >>>>>
> >>>>>
> >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> null
> >>>>> at
> >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at
> >>
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> ~[na:1.8.0_162]
> >>>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> ~[na:1.8.0_162]
> >>>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> ~[na:1.8.0_162]
> >>>>> at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_162]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>> at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>>>
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
Hi
I tried putting the kahadb directory on the local directory, it regenerated on startup but still gave me a BadMagicException
________________________________
From: Jean-Baptiste Onofré <jb...@nanthrax.net>
Sent: Sunday, November 3, 2019 11:07:22 AM
To: users@activemq.apache.org
Subject: Re: org.apache.activemq.kaha.impl.index.?
Hi Jonathan,
user mailing list is fine for this kind of question.
IMHO, I'm pretty sure it's related to the filesystem.
As Tim suggested, maybe you can try to put the kahadb directory on local
filesystem instead of the SAN. It could be an interesting test.
Regards
JB
On 03/11/2019 16:52, Jonathan Roby wrote:
> Hi
>
>
> Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.
>
>
> Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?
>
>
> I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break
>
>
> Thank you everyone for your help so far
>
> Jon
>
> ________________________________
> From: Jonathan Roby
> Sent: Sunday, November 3, 2019 9:02:56 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
>
> Hi
>
>
> I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
>
>
> I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
>
>
> Thank you
>
> Jon
>
> ________________________________
> From: Tim Bain <tb...@alumni.duke.edu>
> Sent: Sunday, November 3, 2019 8:06:29 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Following up on JB's line of questioning, is the directory only recreated
> when (exactly when, to the second, with a matching timestamp) this broker
> process is started? Might it be possible that a different broker process,
> running elsewhere, might be creating the directory and its files?
>
> If you reconfigure this broker to store the KahaDB data file on local disk
> instead of your SAN (just temporarily), does the process start
> successfully?
>
> Tim
>
> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
> wrote:
>
>>
>> No, its not fixed. I wiped out the data folder and the persistant folder.
>> both were regenerated but still is failing
>>
>>
>> its weird as we have other instances set up exactly the same as this one
>> that dont have this error. same versions, same everything
>>
>>
>> so it sounds like you might be correct about it being the filesystem. i
>> hope not as that can be tricky to debug
>>
>>
>> Thank you for your help
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Sent: Saturday, November 2, 2019 12:48:29 PM
>> To: users@activemq.apache.org
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> I guess it's fixed with you complete remove the kahadb folder right ?
>>
>> I remember couple of users mentioned similar issue on virtual machine.
>>
>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>> on this version around kahadb.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>> Hi,
>>>
>>> It's definitely something weird while reading file on the filesystem.
>>>
>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>
>>> Regards
>>> JB
>>>
>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>> Hi
>>>>
>>>>
>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>> now deleted the kahadb directory a few times (it regenerated) and even
>> moved the broker path in case it was a drive error
>>>>
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> Hi,
>>>>
>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>> (and you updated ActiveMQ version later) ?
>>>>
>>>> It seems the kahadb index is not consistent.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>>>
>>>>>
>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>> message passing between components. Yesterday, one of our instances seemed
>> to stop working. after several restarts of components and the system
>> itself, I noticed that when the components restarted, it always produced a
>> nested exception and the very last error was
>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>
>>>>>
>>>>> I've googled this and taken the advice given, which is usually to
>> empty/delete the data directory or the kahadb directory and it still isnt
>> working. All the other components seem to be working fine.
>>>>>
>>>>>
>>>>> Thank you for any help anyone can give me.
>>>>>
>>>>> Jon
>>>>>
>>>>>
>>>>> Here is the last part of the stack trace.
>>>>>
>>>>>
>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> ~[na:1.8.0_162]
>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> ~[na:1.8.0_162]
>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> ~[na:1.8.0_162]
>>>>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Jonathan,
user mailing list is fine for this kind of question.
IMHO, I'm pretty sure it's related to the filesystem.
As Tim suggested, maybe you can try to put the kahadb directory on local
filesystem instead of the SAN. It could be an interesting test.
Regards
JB
On 03/11/2019 16:52, Jonathan Roby wrote:
> Hi
>
>
> Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.
>
>
> Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?
>
>
> I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break
>
>
> Thank you everyone for your help so far
>
> Jon
>
> ________________________________
> From: Jonathan Roby
> Sent: Sunday, November 3, 2019 9:02:56 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
>
> Hi
>
>
> I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
>
>
> I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
>
>
> Thank you
>
> Jon
>
> ________________________________
> From: Tim Bain <tb...@alumni.duke.edu>
> Sent: Sunday, November 3, 2019 8:06:29 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Following up on JB's line of questioning, is the directory only recreated
> when (exactly when, to the second, with a matching timestamp) this broker
> process is started? Might it be possible that a different broker process,
> running elsewhere, might be creating the directory and its files?
>
> If you reconfigure this broker to store the KahaDB data file on local disk
> instead of your SAN (just temporarily), does the process start
> successfully?
>
> Tim
>
> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
> wrote:
>
>>
>> No, its not fixed. I wiped out the data folder and the persistant folder.
>> both were regenerated but still is failing
>>
>>
>> its weird as we have other instances set up exactly the same as this one
>> that dont have this error. same versions, same everything
>>
>>
>> so it sounds like you might be correct about it being the filesystem. i
>> hope not as that can be tricky to debug
>>
>>
>> Thank you for your help
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Sent: Saturday, November 2, 2019 12:48:29 PM
>> To: users@activemq.apache.org
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> I guess it's fixed with you complete remove the kahadb folder right ?
>>
>> I remember couple of users mentioned similar issue on virtual machine.
>>
>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>> on this version around kahadb.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>> Hi,
>>>
>>> It's definitely something weird while reading file on the filesystem.
>>>
>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>
>>> Regards
>>> JB
>>>
>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>> Hi
>>>>
>>>>
>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>> now deleted the kahadb directory a few times (it regenerated) and even
>> moved the broker path in case it was a drive error
>>>>
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>> To: users@activemq.apache.org
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> Hi,
>>>>
>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>> (and you updated ActiveMQ version later) ?
>>>>
>>>> It seems the kahadb index is not consistent.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>>>
>>>>>
>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>> message passing between components. Yesterday, one of our instances seemed
>> to stop working. after several restarts of components and the system
>> itself, I noticed that when the components restarted, it always produced a
>> nested exception and the very last error was
>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>
>>>>>
>>>>> I've googled this and taken the advice given, which is usually to
>> empty/delete the data directory or the kahadb directory and it still isnt
>> working. All the other components seem to be working fine.
>>>>>
>>>>>
>>>>> Thank you for any help anyone can give me.
>>>>>
>>>>> Jon
>>>>>
>>>>>
>>>>> Here is the last part of the stack trace.
>>>>>
>>>>>
>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> ~[na:1.8.0_162]
>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> ~[na:1.8.0_162]
>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> ~[na:1.8.0_162]
>>>>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>> at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
Hi
Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.
Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?
I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break
Thank you everyone for your help so far
Jon
________________________________
From: Jonathan Roby
Sent: Sunday, November 3, 2019 9:02:56 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?
Hi
I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
Thank you
Jon
________________________________
From: Tim Bain <tb...@alumni.duke.edu>
Sent: Sunday, November 3, 2019 8:06:29 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?
Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?
If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?
Tim
On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>> at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
Hi
I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
Thank you
Jon
________________________________
From: Tim Bain <tb...@alumni.duke.edu>
Sent: Sunday, November 3, 2019 8:06:29 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?
Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?
If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?
Tim
On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>> at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Tim Bain <tb...@alumni.duke.edu>.
Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?
If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?
Tim
On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <Jo...@umanitoba.ca>
wrote:
>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: users@activemq.apache.org
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>> at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>> at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
No, its not fixed. I wiped out the data folder and the persistant folder. both were regenerated but still is failing
its weird as we have other instances set up exactly the same as this one that dont have this error. same versions, same everything
so it sounds like you might be correct about it being the filesystem. i hope not as that can be tricky to debug
Thank you for your help
Jon
________________________________
From: Jean-Baptiste Onofré <jb...@nanthrax.net>
Sent: Saturday, November 2, 2019 12:48:29 PM
To: users@activemq.apache.org
Subject: Re: org.apache.activemq.kaha.impl.index.?
I guess it's fixed with you complete remove the kahadb folder right ?
I remember couple of users mentioned similar issue on virtual machine.
Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
on this version around kahadb.
Regards
JB
On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> Hi,
>
> It's definitely something weird while reading file on the filesystem.
>
> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>
> Regards
> JB
>
> On 02/11/2019 18:18, Jonathan Roby wrote:
>> Hi
>>
>>
>> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>>
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Sent: Saturday, November 2, 2019 12:16:31 PM
>> To: users@activemq.apache.org
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi,
>>
>> Does the kahadb folder has been created with the same ActiveMQ version
>> (and you updated ActiveMQ version later) ?
>>
>> It seems the kahadb index is not consistent.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>
>>>
>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>>
>>>
>>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>>
>>>
>>> Thank you for any help anyone can give me.
>>>
>>> Jon
>>>
>>>
>>> Here is the last part of the stack trace.
>>>
>>>
>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
Talend - A Cloud Data Integration Leader (modern ETL)<http://www.talend.com/>
www.talend.com
Talend Data Fabric offers a single suite of cloud apps for data integration and data integrity to help enterprises collect, govern, transform, and share data.
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Talend - A Cloud Data Integration Leader (modern ETL)<http://www.talend.com/>
www.talend.com
Talend Data Fabric offers a single suite of cloud apps for data integration and data integrity to help enterprises collect, govern, transform, and share data.
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I guess it's fixed with you complete remove the kahadb folder right ?
I remember couple of users mentioned similar issue on virtual machine.
Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
on this version around kahadb.
Regards
JB
On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> Hi,
>
> It's definitely something weird while reading file on the filesystem.
>
> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>
> Regards
> JB
>
> On 02/11/2019 18:18, Jonathan Roby wrote:
>> Hi
>>
>>
>> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>>
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Sent: Saturday, November 2, 2019 12:16:31 PM
>> To: users@activemq.apache.org
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi,
>>
>> Does the kahadb folder has been created with the same ActiveMQ version
>> (and you updated ActiveMQ version later) ?
>>
>> It seems the kahadb index is not consistent.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>
>>>
>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>>
>>>
>>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>>
>>>
>>> Thank you for any help anyone can give me.
>>>
>>> Jon
>>>
>>>
>>> Here is the last part of the stack trace.
>>>
>>>
>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
It's definitely something weird while reading file on the filesystem.
Is it a local filesystem or shared one (NFS, CIFS, ...) ?
Regards
JB
On 02/11/2019 18:18, Jonathan Roby wrote:
> Hi
>
>
> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Sent: Saturday, November 2, 2019 12:16:31 PM
> To: users@activemq.apache.org
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi,
>
> Does the kahadb folder has been created with the same ActiveMQ version
> (and you updated ActiveMQ version later) ?
>
> It seems the kahadb index is not consistent.
>
> Regards
> JB
>
> On 02/11/2019 18:14, Jonathan Roby wrote:
>> Hi, i've just joined the list so apologies if i've not this correctly.
>>
>>
>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>
>>
>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>
>>
>> Thank you for any help anyone can give me.
>>
>> Jon
>>
>>
>> Here is the last part of the stack trace.
>>
>>
>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jonathan Roby <Jo...@umanitoba.ca>.
Hi
as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
Jon
________________________________
From: Jean-Baptiste Onofré <jb...@nanthrax.net>
Sent: Saturday, November 2, 2019 12:16:31 PM
To: users@activemq.apache.org
Subject: Re: org.apache.activemq.kaha.impl.index.?
Hi,
Does the kahadb folder has been created with the same ActiveMQ version
(and you updated ActiveMQ version later) ?
It seems the kahadb index is not consistent.
Regards
JB
On 02/11/2019 18:14, Jonathan Roby wrote:
> Hi, i've just joined the list so apologies if i've not this correctly.
>
>
> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>
>
> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>
>
> Thank you for any help anyone can give me.
>
> Jon
>
>
> Here is the last part of the stack trace.
>
>
> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: org.apache.activemq.kaha.impl.index.?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
Does the kahadb folder has been created with the same ActiveMQ version
(and you updated ActiveMQ version later) ?
It seems the kahadb index is not consistent.
Regards
JB
On 02/11/2019 18:14, Jonathan Roby wrote:
> Hi, i've just joined the list so apologies if i've not this correctly.
>
>
> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>
>
> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>
>
> Thank you for any help anyone can give me.
>
> Jon
>
>
> Here is the last part of the stack trace.
>
>
> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
> at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com