You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by William Boutin <wi...@ericsson.com> on 2017/07/19 17:41:10 UTC
Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class
We are running apache-cassandra-2.2.6 for months with no JNA startup issues.
Recently, we have updated some of our cassandra machines and we ran into a Cassandra startup issue with JNA. See the stack trace 1 below.
Question 1: Our Cassandra package contains jna-4.0.0.jar in the /usr/share/cassandra/lib/ directory. Does apache-cassandra-2.2.6 specifically package A-C* with the jna-4.0.0.jar ?
I tried to circumvent the exception by yum installing jna-3.2.4.jar for RHEL 6.4 and adding it to the /usr/share/cassandra/lib/ directory. This got me past the JNA mlockall failure with "INFO 16:56:47 JNA mlockall successful" but when the MemtableFlushWriter requested memory allocation, I now get the STACK TRACE 2 error. "nodetool status" still reports the node as incomplete.
Any Suggestions?
Thank you
STACK TRACE 1:
ERROR [MemtableFlushWriter:1] 2017-07-17 22:33:13,376 CassandraDaemon.java:185 - Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:74) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SafeMemory.<init>(SafeMemory.java:32) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressionMetadata$Writer.<init>(CompressionMetadata.java:304) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:318) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:73) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:75) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:433) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:376) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1146) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_141]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_141]
STACK TRACE 2:
ERROR 16:56:51 Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.NoSuchMethodError: com.sun.jna.Native.malloc(J)J
at org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.Memory.<init>(Memory.java:74) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SafeMemory.<init>(SafeMemory.java:32) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressionMetadata$Writer.<init>(CompressionMetadata.java:304) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:318) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:73) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) ~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:75) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:433) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:376) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1146) ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_141]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_141]
[Ericsson]<http://www.ericsson.com/>
WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE
Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.boutin@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]<http://www.ericsson.com/current_campaign>
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This Communication is Confidential. We only send and receive email on the basis of the terms set out at www.ericsson.com/email_disclaimer<http://www.ericsson.com/email_disclaimer>
Re: RE: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class
Posted by Jeff Jirsa <jj...@apache.org>.
So what precisely changed? You've got a custom build based on the jar name, which is perfectly reasonable, but what upgrade did you do? 2.2.5 to 2.2.6 ? Any other changes?
On 2017-07-20 05:41 (-0700), William Boutin <wi...@ericsson.com> wrote:
> Thank you for your help.
> We have been using jna-4.0.0.jar since using Cassandra 2.2(.6). Until last week, we had no issues. Now, we are experiencing the exception that I identified. We only have jna-4.0.0.jar loaded on our machines and the CLASSPATH that Cassandra builds only uses the jar from the /usr/share/cassandra/lib directory. Jna-3.2.4.jar was only introduced when I was debugging the original issue and will never be loaded on our machines. See the output of ps below.
>
> Should I be looking elsewhere? Some have brought up that low memory on a JVM or LINUX machine could cause jar loading issues. I have tried doubling -Xss256k to -Xss512k.
>
> Thanks
>
> Billy S. Boutin
> Office Phone No. (913) 241-5574
> Cell Phone No. (732) 213-1368
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class
Posted by William Boutin <wi...@ericsson.com>.
Thank you for your help.
We have been using jna-4.0.0.jar since using Cassandra 2.2(.6). Until last week, we had no issues. Now, we are experiencing the exception that I identified. We only have jna-4.0.0.jar loaded on our machines and the CLASSPATH that Cassandra builds only uses the jar from the /usr/share/cassandra/lib directory. Jna-3.2.4.jar was only introduced when I was debugging the original issue and will never be loaded on our machines. See the output of ps below.
Should I be looking elsewhere? Some have brought up that low memory on a JVM or LINUX machine could cause jar loading issues. I have tried doubling -Xss256k to -Xss512k.
Thanks
Billy S. Boutin
Office Phone No. (913) 241-5574
Cell Phone No. (732) 213-1368
496 3318 1 0 Jul19 ? 00:06:20 /usr/java/latest/bin/java -Dorg.xerial.snappy.tempdir=/var/tmp -Djava.io.tmpdir=/var/tmp -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1968M -Xmx1968M -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+PerfDisableSharedMem -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=10000 -XX:+UseCondCardMark -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -Xloggc:/usr/share/cassandra/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 -XX:+DisableExplicitGC -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/usr/share/cassandra/logs -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra/conf:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.2.6-E002.jar:/usr/share/cassandra/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share
/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/crc32ex-0.1.1.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/ohc-core-0.3.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.3.4.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-c
sv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.2.6-E002.jar:/usr/share/cassandra/apache-cassandra-thrift-2.2.6-E002.jar:/usr/share/cassandra/stress.jar: org.apache.cassandra.service.CassandraDaemon
-----Original Message-----
From: Jeff Jirsa [mailto:jjirsa@apache.org]
Sent: Wednesday, July 19, 2017 11:47 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class
On 2017-07-19 10:41 (-0700), William Boutin <wi...@ericsson.com> wrote:
> We are running apache-cassandra-2.2.6 for months with no JNA startup issues.
> Recently, we have updated some of our cassandra machines and we ran into a Cassandra startup issue with JNA. See the stack trace 1 below.
>
> Question 1: Our Cassandra package contains jna-4.0.0.jar in the /usr/share/cassandra/lib/ directory. Does apache-cassandra-2.2.6 specifically package A-C* with the jna-4.0.0.jar ?
>
Yes, we explicitly bundle jna-4.0.0.jar for 2.2.6
This is unchanged in 2.2 since https://github.com/apache/cassandra/commit/2ba394676dc673b6d66a07247dccd122b64b0578 ( Jan 2014, when it was cassandra 2.1 beta).
Is the jar missing? Did another version of jna get installed? If a system package got installed, you may have multiple versions in the classpath.
> I tried to circumvent the exception by yum installing jna-3.2.4.jar for RHEL 6.4 and adding it to the /usr/share/cassandra/lib/ directory. This got me past the JNA mlockall failure with "INFO 16:56:47 JNA mlockall successful" but when the MemtableFlushWriter requested memory allocation, I now get the STACK TRACE 2 error. "nodetool status" still reports the node as incomplete.
>
Installing 3.2.4 is probably not going to work (as hinted in my earlier paragraph); you almost certainly want 4.0 or newer, and you should probably want exactly one jna jar in the classpath at any given time.
- Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-07-19 10:41 (-0700), William Boutin <wi...@ericsson.com> wrote:
> We are running apache-cassandra-2.2.6 for months with no JNA startup issues.
> Recently, we have updated some of our cassandra machines and we ran into a Cassandra startup issue with JNA. See the stack trace 1 below.
>
> Question 1: Our Cassandra package contains jna-4.0.0.jar in the /usr/share/cassandra/lib/ directory. Does apache-cassandra-2.2.6 specifically package A-C* with the jna-4.0.0.jar ?
>
Yes, we explicitly bundle jna-4.0.0.jar for 2.2.6
This is unchanged in 2.2 since https://github.com/apache/cassandra/commit/2ba394676dc673b6d66a07247dccd122b64b0578 ( Jan 2014, when it was cassandra 2.1 beta).
Is the jar missing? Did another version of jna get installed? If a system package got installed, you may have multiple versions in the classpath.
> I tried to circumvent the exception by yum installing jna-3.2.4.jar for RHEL 6.4 and adding it to the /usr/share/cassandra/lib/ directory. This got me past the JNA mlockall failure with "INFO 16:56:47 JNA mlockall successful" but when the MemtableFlushWriter requested memory allocation, I now get the STACK TRACE 2 error. "nodetool status" still reports the node as incomplete.
>
Installing 3.2.4 is probably not going to work (as hinted in my earlier paragraph); you almost certainly want 4.0 or newer, and you should probably want exactly one jna jar in the classpath at any given time.
- Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org