You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Timothy Stewart (JIRA)" <ji...@apache.org> on 2014/06/15 04:34:01 UTC
[jira] [Created] (AMQ-5228) java.lang.NoClassDefFoundError:
org/fusesource/leveldbjni/internal/JniDB error during cleanup
Timothy Stewart created AMQ-5228:
------------------------------------
Summary: java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB error during cleanup
Key: AMQ-5228
URL: https://issues.apache.org/jira/browse/AMQ-5228
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.10.0, 5.9.1
Environment: Linux 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
JDK 1.7.0_51
Apache Karaf 2.3.5 with activemq-osgi, activemq-blueprint, activemq-client, activemq-webconsole, activemq-camel, activemq features installed. Using version 5.9.1 (and tried 5.10.0)
Reporter: Timothy Stewart
In our production environment, our storage folder runs out of disk space every few days (75 Gb). Restarting the container addresses the issue, it cleans it up. I noticed a stack trace coming on system.err in the wrapper.log (Karaf starts with a wrapper service). It may or may not be related to our disk space issue:
INFO | jvm 1 | 2014/06/13 03:22:36 | Exception in thread "Thread-117" java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:377)
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1647)
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.activemq.leveldb.DBManager$$anonfun$pollGc$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(DBManager.scala:648)
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
INFO | jvm 1 | 2014/06/13 03:22:36 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO | jvm 1 | 2014/06/13 03:22:36 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO | jvm 1 | 2014/06/13 03:22:36 | at java.lang.Thread.run(Thread.java:744)
INFO | jvm 1 | 2014/06/13 03:22:36 | Caused by: java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB not found by org.apache.activemq.activemq-osgi [105]
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO | jvm 1 | 2014/06/13 03:22:36 | at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO | jvm 1 | 2014/06/13 03:22:36 | at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO | jvm 1 | 2014/06/13 03:22:36 | ... 7 more
I see the same problem in our dev environment but could not replicate it. I was finally able to replicate by using the hawtio console to execute the compact operation. Everytime I do this, the same stack trace outputs and the operation cycles endlessly (well as long as I've waited anyhow). The 5.10.0 stack trace when I execute the operation is:
INFO | jvm 2 | 2014/06/14 21:00:44 | Exception in thread "Thread-106" java.lang.NoClassDefFoundError: org/fusesource/leveldbjni/internal/JniDB
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.activemq.leveldb.LevelDBClient$RichDB.compact(LevelDBClient.scala:378)
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.activemq.leveldb.LevelDBClient.gc(LevelDBClient.scala:1654)
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.activemq.leveldb.LevelDBStoreView$$anonfun$compact$1.apply$mcV$sp(LevelDBStore.scala:126)
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:330)
INFO | jvm 2 | 2014/06/14 21:00:44 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO | jvm 2 | 2014/06/14 21:00:44 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO | jvm 2 | 2014/06/14 21:00:44 | at java.lang.Thread.run(Thread.java:744)
INFO | jvm 2 | 2014/06/14 21:00:44 | Caused by: java.lang.ClassNotFoundException: org.fusesource.leveldbjni.internal.JniDB not found by org.apache.activemq.activemq-osgi [390]
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:72)
INFO | jvm 2 | 2014/06/14 21:00:44 | at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1843)
INFO | jvm 2 | 2014/06/14 21:00:44 | at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
INFO | jvm 2 | 2014/06/14 21:00:44 | ... 7 more
I started looking at the activemq-osgi and activemq-leveldb project poms, and noticed that the leveldbjni projects are commented out with a note that we want to focus on hardening the pure java version. With that in mind, I switched the indexFactory on my leveldb config to indexFactory="org.iq80.leveldb.impl.Iq80DBFactory". I restarted, and tried the compact operation again, but got the same stacktrace.
My leveldb config now looks like:
<amq:persistenceAdapter>
<amq:levelDB directory="[[karaf.storage]]/activemq/default/leveldb" logSize="107374182" logCompression="snappy" indexFactory="org.iq80.leveldb.impl.Iq80DBFactory" />
</amq:persistenceAdapter>
The logCompression and the indexFactory were both added as an attempt to mitigate the problem, but the issue exists either way. I also saw a note on the replicated levedb store about scheduled jobs, which are used a lot, but when I looked at those folders in our storage folder it seems to use a completely different kahadb store, so I threw that out as a reason.
I did attempt to add leveldbjni-linux64 or leveldbjni-all to the osgi deployment, but it does not export the internal package which activemq-osgi wants to import. I could build a new package of these, or rebuild activemq-osgi with the commented dependencies removed.. perhaps that will work, but it seems that as it is with the dependencies commented out that there is something broken within the compact routine.
--
This message was sent by Atlassian JIRA
(v6.2#6252)