You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Timothy Bish (JIRA)" <ji...@apache.org> on 2012/07/09 21:21:35 UTC
[jira] [Closed] (AMQ-3866) kahadb log files not getting cleaned up
[ https://issues.apache.org/jira/browse/AMQ-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3866.
-----------------------------
Resolution: Incomplete
No further information provided by the user in regards to the suggested solutions.
> kahadb log files not getting cleaned up
> ---------------------------------------
>
> Key: AMQ-3866
> URL: https://issues.apache.org/jira/browse/AMQ-3866
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.5.1, 5.6.0
> Reporter: Alexander Vallens
> Priority: Critical
>
> A few weeks ago we experienced an event where several hundred thousand messages were posted to a queue and slowly consumed over several days until accidentally purged. Ever since then our kahadb has been growing on the order of 12-20 bytes/10 seconds, corresponding to additional entries being added to the kahadb log files that are not being cleaned up (this is even the case in an isolated test environment with no subscribers/consumers/activity). Having searched and found AMQ-3131, AMQ-3736, AMQ-3424, we attempted an upgrade on a clone of the affected system from 5.5.1 to 5.6.0, with no luck. Along the way here are the steps we took:
> 1) I cloned our ActiveMQ virtual machine “hot” (i.e. without shutdown) and was able to get an isolated (different hostname, IP, VLAN), but functioning, “copy” of the production ActiveMQ instance
> 2) startup is slow, given the sheer size of the kahadb, as shown in the startup logging:
> 2012-05-29 19:59:11,780 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,352 | WARN | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:18:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,494 | WARN | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:25:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:28,528 | WARN | Duplicate message add attempt rejected. Destination: Consumer.SchedulerEvents.VirtualTopic.scheduler/events, Message id: ID:<FQDN>-55381-1337386136713-0:32:1:1:1 | org.apache.a
> ctivemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:05:34,513 | INFO | Recovery replayed 2325747 operations from the journal in 382.795 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 3) I was able to eliminate the 3 “duplicate message” warnings by purging the “Consumer.SchedulerEvents.VirtualTopic.scheduler/events” queue (I suspect there were unconsumed messages, perhaps partially uploaded, in that queue at time of clone)
> 4) after the above steps, I was able to observe that kahadb continued to grow at the same rate as the actual production ActiveMQ instancw, ~12-20 bytes every 10s
> 5) during the first normal shutdown of activemq, I observed the following errors:
> java.lang.IllegalStateException: PageFile is not loaded
> at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:715)
> at org.apache.kahadb.page.PageFile.tx(PageFile.java:239)
> at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:510)
> at org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:93)
> at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:85)
> at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.stop(AbstractStoreCursor.java:67)
> at org.apache.activemq.broker.region.cursors.StoreQueueCursor.stop(StoreQueueCursor.java:84)
> at org.apache.activemq.broker.region.Queue.stop(Queue.java:872)
> at org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:110)
> at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:729)
> at org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:113)
> at org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:213)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at org.apache.activemq.broker.scheduler.SchedulerBroker.stop(SchedulerBroker.java:104)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161)
> at org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:114)
> at org.apache.activemq.broker.BrokerService$3.stop(BrokerService.java:1795)
> at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
> at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:587)
> at org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:135)
> 2012-05-29 20:22:09,112 | ERROR | Failed to reset batching | org.apache.activemq.store.kahadb.KahaDBStore | Thread-45
> 6) subsequent shutdowns, however, operated normally without error
> 7) even after various attempts at shutdown and start back up, the aforementioned journal recovery was time consuming:
> 2012-05-29 20:25:06,780 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 20:31:39,821 | INFO | Recovery replayed 2325954 operations from the journal in 393.074 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 8) and the kahadb growth issue persisted
> 9) I went ahead and successfully upgraded to ActiveMQ v5.6.0 (verified queues & messages in scheduled queue survived upgrade)
> 10) first attempt to start activemq after 5.6.0 upgrade included following errors:
> 2012-05-29 20:55:36,825 | ERROR | Failed to start ActiveMQ JMS Message Broker (localhost, null). Reason: java.lang.NegativeArraySizeException | org.apache.activemq.broker.BrokerService | main
> java.lang.NegativeArraySizeException
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
> at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
> at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
> at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
> at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
> at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
> at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
> at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
> at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
> at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
> at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
> at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
> at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
> at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
> at org.apache.activemq.console.Main.main(Main.java:111)
> 2012-05-29 20:55:36,873 | INFO | ActiveMQ Message Broker (localhost, null) is shutting down | org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:36,873 | INFO | Connector openwire Stopped | org.apache.activemq.broker.TransportConnector | main
> 2012-05-29 20:55:36,873 | INFO | Connector stomp Stopped | org.apache.activemq.broker.TransportConnector | main
> 2012-05-29 20:55:36,897 | INFO | PListStore:[/opt/activemq/data/localhost/tmp_storage ] stopped | org.apache.activemq.store.kahadb.plist.PListStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:36,897 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | main
> 2012-05-29 20:55:37,297 | INFO | ActiveMQ JMS Message Broker (localhost, null) stopped | org.apache.activemq.broker.BrokerService | main
> 2012-05-29 20:55:37,300 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invoc
> ation of init method failed; nested exception is java.lang.NegativeArraySizeException | org.apache.activemq.xbean.XBeanBrokerFactory | main
> org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested e
> xception is java.lang.NegativeArraySizeException
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
> at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
> at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
> at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:108)
> at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:108)
> at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:72)
> at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
> at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
> at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
> at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
> at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
> at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:257)
> at org.apache.activemq.console.Main.main(Main.java:111)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:188)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller$1.execute(MessageDatabase.java:1654)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1642)
> at org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller.readPayload(MessageDatabase.java:1627)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:201)
> at org.apache.kahadb.index.BTreeNode$Marshaller.readPayload(BTreeNode.java:147)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:412)
> at org.apache.kahadb.page.Transaction.load(Transaction.java:367)
> at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)
> at org.apache.kahadb.index.BTreeIndex.getRoot(BTreeIndex.java:174)
> at org.apache.kahadb.index.BTreeIndex.iterator(BTreeIndex.java:228)
> at org.apache.activemq.store.kahadb.KahaDBStore$6.execute(KahaDBStore.java:943)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
> at org.apache.activemq.store.kahadb.KahaDBStore.getDestinations(KahaDBStore.java:941)
> at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.getDestinations(KahaDBPersistenceAdapter.java:122)
> at org.apache.activemq.broker.region.DestinationFactoryImpl.getDestinations(DestinationFactoryImpl.java:68)
> at org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2369)
> at org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2198)
> at org.apache.activemq.broker.BrokerService.start(BrokerService.java:527)
> at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1544)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1485)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
> ... 28 more
> 11) after the above errors, ActiveMQ was dead; another attempt to start worked, yielding the following (also logged in previous attempt):
> 2012-05-29 21:58:01,317 | INFO | KahaDB is version 4 | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:01,396 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:27,601 | INFO | @2207:10939359, 100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:42,788 | INFO | @2222:10328575, 200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:58:59,234 | INFO | @2242:14360209, 300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:13,628 | INFO | @2254:1365404, 400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:27,466 | INFO | @2264:30387363, 500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 21:59:43,739 | INFO | @2281:28756842, 600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:00:13,389 | INFO | @2320:26490711, 700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:00:47,350 | INFO | @2371:21837918, 800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:01:17,752 | INFO | @2412:10702462, 900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:01:51,288 | INFO | @2465:32070937, 1000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:02:25,203 | INFO | @2519:21797589, 1100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:02:55,188 | INFO | @2564:13289299, 1200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:05,786 | INFO | @2572:23712519, 1300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:15,375 | INFO | @2578:22618226, 1400000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:27,246 | INFO | @2585:31399538, 1500000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:38,213 | INFO | @2593:7102179, 1600000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:48,285 | INFO | @2600:11040885, 1700000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:03:55,053 | INFO | @2602:28573821, 1800000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:01,331 | INFO | @2603:32850602, 1900000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:06,689 | INFO | @2605:3582539, 2000000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:12,435 | INFO | @2606:7920235, 2100000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:17,500 | INFO | @2607:12248010, 2200000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:25,521 | INFO | @2617:28414362, 2300000 entries recovered .. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 2012-05-29 22:04:28,459 | INFO | Recovery replayed 2326036 operations from the journal in 387.094 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
> 12) unfortunately, I confirmed that even after upgrade, kahadb continues to grow at about the same rate as before
> 13) I then applied the logging parameters as described in http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html. . With debugging spun up we see A LOT (I can’t get a count because log files rotate too fast, but in the 6 logs that remain I get 31,978 entries) of the following:
> 2012-05-29 23:18:41,925 | DEBUG | message not found in sequence id index: ID:<FQDN>-59596-1337475432262-0:35:1:1:1202 | org.apache.activemq.store.kahadb.MessageDatabase | main
> Google and a search on issues.apache.org reveal nothing on this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira