[jira] [Created] (AMQ-6282) LevelDB store can't recover from no disk space

["Ryan Stedman (JIRA)" <ji...@apache.org>]

Ryan Stedman created AMQ-6282:
---------------------------------

             Summary: LevelDB store can't recover from no disk space
                 Key: AMQ-6282
                 URL: https://issues.apache.org/jira/browse/AMQ-6282
             Project: ActiveMQ
          Issue Type: Bug
          Components: activemq-leveldb-store
    Affects Versions: 5.12.2
         Environment: CentOS 6.5
            Reporter: Ryan Stedman


If there is no space left on the drive that leveldb is using for storage, it will wind up truncating the store-version.txt file, which will prevent activemq from starting up once the drive space issue is resolved.

To reproduce:
1. Start activemq, configured to use leveldb as a persistence store.
2. use fallocate to create a file(s) that use up all available disk space on the drive that leveldb is using to persist to.
3. restart activemq.  It will fail to start with errors in the log about no available disk space (expected).
4. delete the files created in step 2, so that the drive is not full
5. start activemq again.  This time it will fail with the below exception:

2016-05-05 13:34:39,986 [erSimpleAppMain] ERROR XBeanBrokerFactory             - Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.Exception: Cannot open the store.  It's schema version is not supported.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.lang.Exception: Cannot open the store.  It's schema version is not supported.
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:934)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
        at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
        at org.apache.activemq.console.Main.main(Main.java:115)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Cannot open the store.  It's schema version is not supported.
        at org.apache.activemq.leveldb.LevelDBClient.init(LevelDBClient.scala:596)
        at org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:557)
        at org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
        at org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
        at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:651)
        at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:640)
        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:604)
        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
        ... 33 more


6. If you look at store-version.txt file (ie. /usr/share/activemq/data/leveldb/store-version.txt), the file is empty.
7. If you edit the file and add the text "activemq_leveldb_store:1" to it, activemq will start up successfully the next time you try to start it.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)