You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by RuralHunter <ru...@gmail.com> on 2010/03/25 02:26:13 UTC

ActiveMQ failed to start after server crash

Hi,
I'm using kahadb as storage. I have seen this several times, for both 5.3.0
and 5.3.1. If there are many messages in queues and ActiveMQ is not stopped
in a correct manner, such as server crash or reboot accidentally, then
ActiveMQ can not be started. I have to delete all files in data directory to
make it start-able. But this kind of operation causes data loss. Is there
anyway to fix that? Here is the log when it fails to start:
2010-03-23 21:28:19,187 | INFO  | ActiveMQ 5.3.1 JMS Message Broker
(datacenter) is starting | org.apache.activemq.broker.BrokerService | main
2010-03-23 21:28:19,187 | INFO  | For help or more information please see:
http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService |
main
2010-03-23 21:28:19,937 | INFO  | Kaha Store using data directory
D:\Application\apache-activemq-5.3.1\bin\..\data\datacenter\tmp_storage |
org.apache.activemq.kaha.impl.KahaStore | main
2010-03-23 21:28:21,687 | ERROR | Failed to start ActiveMQ JMS Message
Broker. Reason: java.lang.NullPointerException |
org.apache.activemq.broker.BrokerService | main
java.lang.NullPointerException
	at
org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:109)
	at
org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:120)
	at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:222)
	at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
	at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:219)
	at org.apache.activemq.broker.region.Queue.initialize(Queue.java:251)
	at
org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
	at
org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:464)
	at
org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
	at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:120)
	at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:268)
	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
	at
org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:153)
	at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
	at
org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:149)
	at
org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:94)
	at
org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:178)
	at
org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:100)
	at
org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:112)
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
	at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
	at
org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1707)
	at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)
	at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
	at java.security.AccessController.doPrivileged(Native Method)
	at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
	at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
	at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
	at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
	at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
	at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
	at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
	at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
	at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
	at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
	at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:96)
	at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:52)
	at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
	at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
	at
org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
	at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
	at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
	at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)
	at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
	at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
	at org.apache.activemq.console.Main.main(Main.java:107)
2010-03-23 21:28:21,781 | INFO  | ActiveMQ Message Broker (datacenter, null)
is shutting down | org.apache.activemq.broker.BrokerService | main
2010-03-23 21:28:21,781 | INFO  | Connector openwire Stopped |
org.apache.activemq.broker.TransportConnector | main
2010-03-23 21:28:22,843 | INFO  | ActiveMQ JMS Message Broker (datacenter,
null) stopped | org.apache.activemq.broker.BrokerService | main
-- 
View this message in context: http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28023136.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ failed to start after server crash

Posted by RuralHunter <ru...@gmail.com>.
Hi Gary,

Thanks for guidance. I opened AMQ-2672 and compressed/uploaded the data:
https://issues.apache.org/activemq/browse/AMQ-2672


Gary Tully wrote:
> 
> For detail of the jira see:
> http://activemq.apache.org/contributing.html#Contributing-Ifyoufindabugorproblem
> 
> 200MB may be too big for confluence (jira) also and also for gmail(25MB),
> maybe try one of the fileshare sites, not sure what to recommend here.
> 
> On 25 March 2010 13:23, RuralHunter <ru...@gmail.com> wrote:
> 
>>
>> How to? I haven't used jira before...and the data directory is quite
>> large,
>> more than200M.
>>
>>
>> Gary Tully wrote:
>> >
>> > Would it be possible to tar and compress a data directory that fails to
>> > start and attach it to a jira issue?
>> >
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28028802.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 

-- 
View this message in context: http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28037820.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ failed to start after server crash

Posted by Gary Tully <ga...@gmail.com>.
For detail of the jira see:
http://activemq.apache.org/contributing.html#Contributing-Ifyoufindabugorproblem

200MB may be too big for confluence (jira) also and also for gmail(25MB),
maybe try one of the fileshare sites, not sure what to recommend here.

On 25 March 2010 13:23, RuralHunter <ru...@gmail.com> wrote:

>
> How to? I haven't used jira before...and the data directory is quite large,
> more than200M.
>
>
> Gary Tully wrote:
> >
> > Would it be possible to tar and compress a data directory that fails to
> > start and attach it to a jira issue?
> >
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28028802.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: ActiveMQ failed to start after server crash

Posted by RuralHunter <ru...@gmail.com>.
How to? I haven't used jira before...and the data directory is quite large,
more than200M.


Gary Tully wrote:
> 
> Would it be possible to tar and compress a data directory that fails to
> start and attach it to a jira issue?
> 
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 

-- 
View this message in context: http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28028802.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ failed to start after server crash

Posted by Gary Tully <ga...@gmail.com>.
Would it be possible to tar and compress a data directory that fails to
start and attach it to a jira issue?

On 25 March 2010 01:26, RuralHunter <ru...@gmail.com> wrote:

>
> Hi,
> I'm using kahadb as storage. I have seen this several times, for both 5.3.0
> and 5.3.1. If there are many messages in queues and ActiveMQ is not stopped
> in a correct manner, such as server crash or reboot accidentally, then
> ActiveMQ can not be started. I have to delete all files in data directory
> to
> make it start-able. But this kind of operation causes data loss. Is there
> anyway to fix that? Here is the log when it fails to start:
> 2010-03-23 21:28:19,187 | INFO  | ActiveMQ 5.3.1 JMS Message Broker
> (datacenter) is starting | org.apache.activemq.broker.BrokerService | main
> 2010-03-23 21:28:19,187 | INFO  | For help or more information please see:
> http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService |
> main
> 2010-03-23 21:28:19,937 | INFO  | Kaha Store using data directory
> D:\Application\apache-activemq-5.3.1\bin\..\data\datacenter\tmp_storage |
> org.apache.activemq.kaha.impl.KahaStore | main
> 2010-03-23 21:28:21,687 | ERROR | Failed to start ActiveMQ JMS Message
> Broker. Reason: java.lang.NullPointerException |
> org.apache.activemq.broker.BrokerService | main
> java.lang.NullPointerException
>        at
>
> org.apache.kahadb.index.BTreeNode$BTreeIterator.findNextPage(BTreeNode.java:109)
>        at
> org.apache.kahadb.index.BTreeNode$BTreeIterator.hasNext(BTreeNode.java:120)
>        at
>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:222)
>        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
>        at
>
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:219)
>        at
> org.apache.activemq.broker.region.Queue.initialize(Queue.java:251)
>        at
>
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>        at
>
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:464)
>        at
>
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>        at
>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:120)
>        at
>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:268)
>        at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
>        at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
>        at
>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:153)
>        at
>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:142)
>        at
>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:149)
>        at
>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:94)
>        at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:178)
>        at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:100)
>        at
>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:112)
>        at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
>        at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
>        at
> org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1707)
>        at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)
>        at
>
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>        at java.lang.reflect.Method.invoke(Unknown Source)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
>        at
>
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
>        at
>
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
>        at
>
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
>        at
>
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>        at
>
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>        at
>
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
>        at
>
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
>        at
>
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
>        at
>
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
>        at
>
> org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
>        at
>
> org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:96)
>        at
>
> org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:52)
>        at
>
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
>        at
>
> org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
>        at
>
> org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
>        at
>
> org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
>        at
>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>        at
>
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)
>        at
>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
>        at
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>        at java.lang.reflect.Method.invoke(Unknown Source)
>        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
>        at org.apache.activemq.console.Main.main(Main.java:107)
> 2010-03-23 21:28:21,781 | INFO  | ActiveMQ Message Broker (datacenter,
> null)
> is shutting down | org.apache.activemq.broker.BrokerService | main
> 2010-03-23 21:28:21,781 | INFO  | Connector openwire Stopped |
> org.apache.activemq.broker.TransportConnector | main
> 2010-03-23 21:28:22,843 | INFO  | ActiveMQ JMS Message Broker (datacenter,
> null) stopped | org.apache.activemq.broker.BrokerService | main
> --
> View this message in context:
> http://old.nabble.com/ActiveMQ-failed-to-start-after-server-crash-tp28023136p28023136.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com