You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by harishkumar <ch...@gmail.com> on 2015/07/08 18:16:19 UTC
Unable to restart the activeMQ with the error "Chunk stream doesn't
exist,Page is marked for free"
Hi All,
Recently we have faced an issue where activeMQ process is shutting down and
not able to start the process. We are using "master/slave" brokers and
kahadb as a message storage.
We have found the "corrupt journal files" error message in activemq.log
(Attached the screen shot). We have applied the below changes to ignore the
corrupted journal files.
<http://activemq.2283324.n4.nabble.com/file/n4698805/corrupt.png>
activemq.xml
<http://activemq.2283324.n4.nabble.com/file/n4698805/activemq.xml>
<persistenceAdapter>
<kahaDB directory="/data02/apphome/Broker" journalMaxFileLength="100mb"
cleanupInterval="5000" ignoreMissingJournalfiles="true"
checkForCorruptJournalFiles="true"
checksumJournalFiles="true"/>
</persistenceAdapter/>
But after that, we are not able to restart the activeMQ process, we found
the below exception in activemq.log file.
2015-05-18 11:26:47,900 | INFO | Connector clientconnection stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2015-05-18 11:26:47,937 | INFO |
JobSchedulerStore:/data02/apphome/Broker/scheduler stopped |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | ActiveMQ
ShutdownHook
2015-05-18 11:26:47,951 | INFO | Broker Plugin
org.apache.activemq.broker.util.RedeliveryPlugin stopped |
org.apache.activemq.broker.BrokerPluginSupport | ActiveMQ ShutdownHook
2015-05-18 11:26:47,951 | INFO |
PListStore:[/data02/apphome/Broker/tmp_storage] stopped |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ
ShutdownHook
2015-05-18 11:26:47,952 | INFO | Stopping async queue tasks |
org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
2015-05-18 11:26:47,952 | INFO | Stopping async topic tasks |
org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
2015-05-18 11:26:47,952 | INFO | Stopped KahaDB |
org.apache.activemq.store.kahadb.KahaDBStore | ActiveMQ ShutdownHook
2015-05-18 11:26:48,139 | INFO | Apache ActiveMQ 5.9.0 (Broker,
ID:deherwsvlws154.wsint.biz-41364-1431682528061-1:1) uptime 2 days 23 hours
| org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2015-05-18 11:26:48,139 | INFO | Apache ActiveMQ 5.9.0 (Broker,
ID:deherwsvlws154.wsint.biz-41364-1431682528061-1:1) is shutdown |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2015-05-18 11:27:06,787 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@5999da01: startup date [Mon
May 18 11:27:06 CEST 2015]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2015-05-18 11:27:08,659 | INFO |
PListStore:[/data02/apphome/Broker/tmp_storage] started |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain
2015-05-18 11:27:09,364 | INFO |
JobSchedulerStore:/data02/apphome/Broker/scheduler started |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
WrapperSimpleAppMain
2015-05-18 11:27:09,365 | INFO | JobScheduler using directory:
/data02/apphome/Broker/scheduler | org.apache.activemq.broker.BrokerService
| WrapperSimpleAppMain
2015-05-18 11:27:09,508 | INFO | Using Persistence Adapter:
KahaDBPersistenceAdapter[/data02/apphome/Broker] |
org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2015-05-18 11:27:09,548 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2015-05-18 11:27:47,926 | INFO | Corrupt journal records found in
'/data02/apphome/Broker/db-21259.log' between offsets: 36319655..36319985 |
org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain
2015-05-18 11:27:49,958 | INFO | Corrupt journal records found in
'/data02/apphome/Broker/db-21260.log' between offsets: 97434148..97606183 |
org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain
2015-05-18 11:27:50,477 | INFO | Corrupt journal records found in
'/data02/apphome/Broker/db-21261.log' between offsets: 37403738..37533983 |
org.apache.activemq.store.kahadb.disk.journal.Journal | WrapperSimpleAppMain
2015-05-18 11:28:06,946 | INFO | Corrupt journal records found in
'/data02/apphome/Broker/db-21273.log' between offsets: 104495634..104647715
| org.apache.activemq.store.kahadb.disk.journal.Journal |
WrapperSimpleAppMain
2015-05-18 11:28:13,519 | INFO | KahaDB is version 5 |
org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
2015-05-18 11:28:13,793 | INFO | Recovering from the journal ... |
org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
2015-05-18 11:28:13,795 | INFO | Recovery replayed 3 operations from the
journal in 0.036 seconds. | org.apache.activemq.store.kahadb.MessageDatabase
| WrapperSimpleAppMain
2015-05-18 11:28:13,814 | ERROR | Failed to start Apache ActiveMQ ([Broker,
null], java.io.EOFException: Chunk stream does not exist, page: 19101 is
marked free) | org.apache.activemq.broker.BrokerService |
WrapperSimpleAppMain
2015-05-18 11:28:13,818 | INFO | Apache ActiveMQ 5.9.0 (Broker, null) is
shutting down | org.apache.activemq.broker.BrokerService |
WrapperSimpleAppMain
2015-05-18 11:28:13,818 | INFO | Connector clientconnection stopped |
org.apache.activemq.broker.TransportConnector | WrapperSimpleAppMain
2015-05-18 11:28:13,834 | INFO |
JobSchedulerStore:/data02/apphome/Broker/scheduler stopped |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
WrapperSimpleAppMain
2015-05-18 11:28:13,834 | INFO |
PListStore:[/data02/apphome/Broker/tmp_storage] stopped |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | WrapperSimpleAppMain
2015-05-18 11:28:13,835 | INFO | Stopping async queue tasks |
org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain
2015-05-18 11:28:13,835 | INFO | Stopping async topic tasks |
org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain
2015-05-18 11:28:13,835 | INFO | Stopped KahaDB |
org.apache.activemq.store.kahadb.KahaDBStore | WrapperSimpleAppMain
2015-05-18 11:28:14,269 | INFO | Apache ActiveMQ 5.9.0 (Broker, null)
uptime 1 minute | org.apache.activemq.broker.BrokerService |
WrapperSimpleAppMain
2015-05-18 11:28:14,269 | INFO | Apache ActiveMQ 5.9.0 (Broker, null) is
shutdown | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2015-05-18 11:28:14,274 | ERROR | Failed to load: class path resource
[activemq.xml], reason: Error creating bean with name
'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path
resource [activemq.xml]: Invocation of init method failed; nested exception
is java.io.EOFException: Chunk stream does not exist, page: 19101 is marked
free | org.apache.activemq.xbean.XBeanBrokerFactory | WrapperSimpleAppMain
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class
path resource [activemq.xml]: Invocation of init method failed; nested
exception is java.io.EOFException: Chunk stream does not exist, page: 19101
is marked free
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1482)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.EOFException: Chunk stream does not exist, page: 19101 is
marked free
at
org.apache.activemq.store.kahadb.disk.page.Transaction$2.readPage(Transaction.java:470)
at
org.apache.activemq.store.kahadb.disk.page.Transaction$2.<init>(Transaction.java:447)
at
org.apache.activemq.store.kahadb.disk.page.Transaction.openInputStream(Transaction.java:444)
at
org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:420)
at
org.apache.activemq.store.kahadb.disk.page.Transaction.load(Transaction.java:377)
at
org.apache.activemq.store.kahadb.disk.index.BTreeIndex.loadNode(BTreeIndex.java:262)
at
org.apache.activemq.store.kahadb.disk.index.BTreeNode.getChild(BTreeNode.java:225)
at
org.apache.activemq.store.kahadb.disk.index.BTreeNode.visit(BTreeNode.java:592)
at
org.apache.activemq.store.kahadb.disk.index.BTreeIndex.visit(BTreeIndex.java:236)
at
org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:744)
at
org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:602)
at
org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)
at
org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:599)
at
org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:389)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:407)
at
org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:260)
at
org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:194)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:215)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:613)
at
org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:602)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:567)
at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1608)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1549)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479)
... 33 more
As this is very critical for us, we deleted the kahadb and restarted the
activeMQ, it started working fine.
We have gone through some of the cases in activemq issue tracker.
https://issues.apache.org/jira/browse/AMQ-2935
This issue has been fixed in ActiveMQ 5.4.2. As per the solution in the
ticket, it is not possible for me to make "SchedulerSupport" as false, which
breaks redelievery policy.
Initially we faced this issue in activeMQ 5.9 version. Recently we have
upgraded activeMQ version 5.11.0 and still issue is exists in latest version
also.
I have attached activemq.xml file.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Unable-to-restart-the-activeMQ-with-the-error-Chunk-stream-doesn-t-exist-Page-is-marked-for-free-tp4698805.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.