You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dustin Wenz <du...@ebureau.com> on 2012/06/07 19:51:09 UTC
Cassandra 1.1.1 crash during compaction
We observed a JRE crash on one node in a seven node cluster about a half hour after upgrading to version 1.1.1 yesterday. Immediately after the upgrade, everything seemed to be working fine. The last item in the cassandra log was a info-level notification that compaction had started on a data file. Four minutes later, the process crashed.
The host OS is FreeBSD 8.2, built for the amd64 architecture. Most of the cluster settings are left to their defaults and the replication factor is set to 2 for our keyspace. We are using the RandomPartitioner and RackInferringSnitch. JNA is enabled, but cannot use mlockall since the process runs as a non-privileged user. It was also necessary to build our own Snappy compressor jar file, since the required architecture was not built-in to the public distribution.
Cassandra is a fairly new software deployment for us, and I was hoping someone could give me some pointers on interpreting the crash report below.
Thanks,
- .Dustin
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x0000000801199140, pid=44897, tid=0x8d1fdc80
#
# Java VM: Diablo Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode bsd-amd64)
# Problematic frame:
# V [libjvm.so+0x599140]
#
# Please submit bug reports to freebsd-java@FreeBSD.org
#
--------------- T H R E A D ---------------
Current thread (0x0000000aa9d56000): JavaThread "CompactionExecutor:30" daemon [_thread_in_vm, id=-1927291776, stack(0x00007ffff898e000,0x00007ffff8a8e000)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=3 (BUS_OBJERR), si_addr=0x0000000801199140
Registers:
RAX=0x0000000aa95f8fe8, RBX=0x0000000aa95fc2c0, RCX=0x0000000987a334f0, RDX=0x00000009274e9888
RSP=0x00007ffff8a8d630, RBP=0x00007ffff8a8d640, RSI=0x00000009274e9888, RDI=0xc900000009274e99
R8 =0x000000098c203bd8, R9 =0x00000008809ff4b8, R10=0x0000000801488580, R11=0x0000000000000001
R12=0x0000000aa9d56000, R13=0x0000000aa95f8c00, R14=0x00007ffff8a8d818, R15=0x0000000aa95f8c10
RIP=0x0000000801199140, EFL=0x003b003b00000001, ERR=0x0000000000000000
TRAPNO=0x001b001300000009
Top of Stack: (sp=0x00007ffff8a8d630)
0x00007ffff8a8d630: 0000000aa95fc2c0 0000000aa9d56000
0x00007ffff8a8d640: 00007ffff8a8d660 000000080119917e
0x00007ffff8a8d650: 000000080345b690 0000000aa95fc2c0
0x00007ffff8a8d660: 00007ffff8a8d6a0 0000000800f3a1fd
0x00007ffff8a8d670: 0000000aa95f8fe8 00000009274e9869
0x00007ffff8a8d680: 0000000986f7acb0 0000000986f7e591
0x00007ffff8a8d690: 00007ffff8a8d818 0000000aa9d56000
0x00007ffff8a8d6a0: 00007ffff8a8d700 000000080346556f
0x00007ffff8a8d6b0: 00000009274e9888 000000080346553b
0x00007ffff8a8d6c0: 00007ffff8a8d6c0 0000000986f7e591
0x00007ffff8a8d6d0: 00007ffff8a8d818 0000000987a30198
0x00007ffff8a8d6e0: 0000000987a334f0 0000000986f7e918
0x00007ffff8a8d6f0: 0000000000000000 00007ffff8a8d810
0x00007ffff8a8d700: 00007ffff8a8d870 000000080345c04e
0x00007ffff8a8d710: 0000000000000000 0000000000000000
0x00007ffff8a8d720: 0000000000000000 0000000000000000
0x00007ffff8a8d730: 0000000000000000 0000000000000000
0x00007ffff8a8d740: 00000009274aa428 000000080817df80
0x00007ffff8a8d750: 00000008081714c0 0000000000000000
0x00007ffff8a8d760: 000000080817df30 0000000000000000
0x00007ffff8a8d770: 000000091b5ed0a0 0000000000000000
0x00007ffff8a8d780: 0000000000000000 0000000000000000
0x00007ffff8a8d790: 0000000000000000 0000000000000000
0x00007ffff8a8d7a0: 0000000000000000 0000000000000000
0x00007ffff8a8d7b0: 0000000000000000 0000000000000000
0x00007ffff8a8d7c0: 0000000000000000 0000000000000000
0x00007ffff8a8d7d0: 0000000000000000 000000000017974b
0x00007ffff8a8d7e0: deaddeaf00000000 00000137c3fdd558
0x00007ffff8a8d7f0: deaddeaf00000000 0000000000000000
0x00007ffff8a8d800: 0000000000000000 000000091b638910
0x00007ffff8a8d810: 0000000000000000 0000000000000000
0x00007ffff8a8d820: 0000000882d0d2e8 00000008406a3de8
Instructions: (pc=0x0000000801199140)
0x0000000801199130: 55 48 89 e5 48 89 5d f0 4c 89 65 f8 48 83 ec 10
0x0000000801199140: 0f b7 47 10 48 89 fb 44 8d 60 01 49 63 fc e8 4d
Stack: [0x00007ffff898e000,0x00007ffff8a8e000], sp=0x00007ffff8a8d630, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x599140]
V [libjvm.so+0x59917e]
V [libjvm.so+0x33a1fd]
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
v ~BufferBlob::Interpreter
J java.util.concurrent.ThreadPoolExecutor$Worker.run()V
v ~BufferBlob::Interpreter
v ~BufferBlob::StubRoutines (1)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
=>0x0000000aa9d56000 JavaThread "CompactionExecutor:30" daemon [_thread_in_vm, id=-1927291776, stack(0x00007ffff898e000,0x00007ffff8a8e000)]
0x00000009962e8000 JavaThread "CompactionExecutor:29" daemon [_thread_in_native, id=-1927291328, stack(0x00007ffff787d000,0x00007ffff797d000)]
0x0000000aa6070400 JavaThread "Thread-58" [_thread_blocked, id=-1823155584, stack(0x00007ffffc7cc000,0x00007ffffc8cc000)]
0x0000000aa7f1c400 JavaThread "Streaming to /172.16.1.17:1" daemon [_thread_in_native, id=-1509346816, stack(0x00007ffffbfc4000,0x00007ffffc0c4000)]
0x0000000aa7c37000 JavaThread "NonPeriodicTasks:1" daemon [_thread_blocked, id=-1927303872, stack(0x00007ffff8085000,0x00007ffff8185000)]
0x0000000aa9bf2000 JavaThread "MemoryMeter:1" daemon [_thread_blocked, id=-1823136768, stack(0x00007ffffcacf000,0x00007ffffcbcf000)]
0x0000000aa95c8c00 JavaThread "CompactionExecutor:24" daemon [_thread_in_Java, id=-1806730048, stack(0x00007ffff757a000,0x00007ffff767a000)]
0x0000000aa6074000 JavaThread "Thread-50" [_thread_blocked, id=-1823156032, stack(0x00007ffffb0b5000,0x00007ffffb1b5000)]
0x0000000aa6077800 JavaThread "Thread-42" [_thread_in_native, id=-1927310144, stack(0x00007ffff868b000,0x00007ffff878b000)]
0x0000000aa9422000 JavaThread "AntiEntropySessions:5" daemon [_thread_blocked, id=-1927308352, stack(0x00007ffffadb2000,0x00007ffffaeb2000)]
0x0000000aa7f1cc00 JavaThread "Streaming to /172.16.1.12:1" daemon [_thread_in_native, id=-1927309248, stack(0x00007ffffcdd2000,0x00007ffffced2000)]
0x0000000aa69e0400 JavaThread "Thrift:16" daemon [_thread_blocked, id=-1806731840, stack(0x00007ffffbdc2000,0x00007ffffbec2000)]
0x0000000aa69e0800 JavaThread "Thrift:15" daemon [_thread_blocked, id=-1806729152, stack(0x00007ffff767b000,0x00007ffff777b000)]
0x0000000aa6078400 JavaThread "Thread-39" [_thread_in_native, id=-1806731392, stack(0x00007ffff777c000,0x00007ffff787c000)]
0x0000000aa9423c00 JavaThread "AntiEntropySessions:1" daemon [_thread_blocked, id=-1823156928, stack(0x00007ffffc5ca000,0x00007ffffc6ca000)]
0x0000000aa9424400 JavaThread "JMX server connection timeout 432" daemon [_thread_blocked, id=-1509345920, stack(0x00007ffff7c81000,0x00007ffff7d81000)]
0x0000000aa69e1000 JavaThread "Thrift:14" daemon [_thread_blocked, id=-1823155136, stack(0x00007ffffaeb3000,0x00007ffffafb3000)]
0x0000000994701400 JavaThread "RMI TCP Connection(34)-172.16.1.11" daemon [_thread_blocked, id=-1823140800, stack(0x00007ffff858a000,0x00007ffff868a000)]
0x00000009949b2c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=-1823152000, stack(0x00007ffff8489000,0x00007ffff8589000)]
0x0000000aa69e1800 JavaThread "Thrift:13" daemon [_thread_blocked, id=-1823151104, stack(0x00007ffff7a7f000,0x00007ffff7b7f000)]
0x0000000aa6078c00 JavaThread "Thread-35" [_thread_in_native, id=-1823133632, stack(0x00007ffff878c000,0x00007ffff888c000)]
0x0000000aa69e2000 JavaThread "Thrift:12" daemon [_thread_blocked, id=-1806733184, stack(0x00007ffffacb1000,0x00007ffffadb1000)]
0x0000000aa6079800 JavaThread "Thread-30" [_thread_in_native, id=-1806732736, stack(0x00007ffff8388000,0x00007ffff8488000)]
0x0000000aa69e2400 JavaThread "Thrift:11" daemon [_thread_blocked, id=-1823157824, stack(0x00007ffff8287000,0x00007ffff8387000)]
0x0000000aa69e2c00 JavaThread "Thrift:10" daemon [_thread_in_native, id=-1823153792, stack(0x00007ffffabb0000,0x00007ffffacb0000)]
0x0000000aa69e3400 JavaThread "Thrift:9" daemon [_thread_blocked, id=-1823153344, stack(0x00007ffffaaaf000,0x00007ffffabaf000)]
0x0000000aa69e3c00 JavaThread "Thrift:8" daemon [_thread_blocked, id=-1823141248, stack(0x00007ffff090e000,0x00007ffff0a0e000)]
0x0000000aa69e4000 JavaThread "Thrift:7" daemon [_thread_blocked, id=-1927307904, stack(0x00007ffffbcc1000,0x00007ffffbdc1000)]
0x0000000aa607a800 JavaThread "Thread-22" [_thread_in_native, id=-1927307456, stack(0x00007ffffb8bd000,0x00007ffffb9bd000)]
0x0000000aa69e4800 JavaThread "Thrift:6" daemon [_thread_blocked, id=-1927306560, stack(0x00007ffffb5ba000,0x00007ffffb6ba000)]
0x0000000aa69e5000 JavaThread "Thrift:5" daemon [_thread_blocked, id=-1927305664, stack(0x00007ffffb3b8000,0x00007ffffb4b8000)]
0x0000000aa69e5800 JavaThread "Thrift:4" daemon [_thread_blocked, id=-1927304320, stack(0x00007ffffcbd0000,0x00007ffffccd0000)]
0x0000000aa69e5c00 JavaThread "Thrift:3" daemon [_thread_blocked, id=-1498747776, stack(0x00007ffffb7bc000,0x00007ffffb8bc000)]
0x0000000aa69e6400 JavaThread "Thrift:2" daemon [_thread_blocked, id=-1509348160, stack(0x00007ffff0b10000,0x00007ffff0c10000)]
0x0000000aa69e6c00 JavaThread "Thrift:1" daemon [_thread_blocked, id=-1509347264, stack(0x00007ffff080d000,0x00007ffff090d000)]
0x0000000801aa2000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10530368, stack(0x00007fffffaff000,0x00007fffffbff000)]
0x000000099452b000 JavaThread "Thread-16" [_thread_in_native, id=-1806747968, stack(0x00007fffeff04000,0x00007ffff0004000)]
0x000000099452b800 JavaThread "Thread-15" [_thread_in_native, id=-1806748416, stack(0x00007ffff0005000,0x00007ffff0105000)]
0x0000000aa607b400 JavaThread "Thread-13" [_thread_in_native, id=-1509348608, stack(0x00007ffff0409000,0x00007ffff0509000)]
0x00000009bdcb5400 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=-1115419520, stack(0x00007ffff060b000,0x00007ffff070b000)]
0x0000000aa607c400 JavaThread "Thread-11" [_thread_in_native, id=-1509347712, stack(0x00007ffff070c000,0x00007ffff080c000)]
0x0000000aa61c3000 JavaThread "WRITE-/172.16.1.14" daemon [_thread_blocked, id=-1507987840, stack(0x00007ffff0c11000,0x00007ffff0d11000)]
0x0000000aa61c3800 JavaThread "WRITE-/172.16.1.14" daemon [_thread_blocked, id=-1507987392, stack(0x00007ffff0d12000,0x00007ffff0e12000)]
0x0000000aa61c4000 JavaThread "WRITE-/172.16.1.13" daemon [_thread_blocked, id=-1509345024, stack(0x00007ffff1419000,0x00007ffff1519000)]
0x0000000aa61c4400 JavaThread "WRITE-/172.16.1.13" daemon [_thread_blocked, id=-1509344576, stack(0x00007ffff171c000,0x00007ffff181c000)]
0x0000000aa61c4c00 JavaThread "WRITE-/172.16.1.16" daemon [_thread_blocked, id=-1509344128, stack(0x00007ffff181d000,0x00007ffff191d000)]
0x0000000aa61c5400 JavaThread "WRITE-/172.16.1.16" daemon [_thread_blocked, id=27571072, stack(0x00007ffffced3000,0x00007ffffcfd3000)]
0x0000000aa607f000 JavaThread "Thread-5" [_thread_in_native, id=-1509345472, stack(0x00007ffff0f14000,0x00007ffff1014000)]
0x0000000aa61c5c00 JavaThread "WRITE-/172.16.1.17" daemon [_thread_blocked, id=-1507986944, stack(0x00007ffff1015000,0x00007ffff1115000)]
0x0000000aa61c6000 JavaThread "WRITE-/172.16.1.17" daemon [_thread_blocked, id=-1507986496, stack(0x00007ffff1116000,0x00007ffff1216000)]
0x0000000aa61c6800 JavaThread "WRITE-/172.16.1.15" daemon [_thread_blocked, id=-1507986048, stack(0x00007ffff1217000,0x00007ffff1317000)]
0x0000000aa61c7000 JavaThread "WRITE-/172.16.1.15" daemon [_thread_blocked, id=-1507985600, stack(0x00007ffff1318000,0x00007ffff1418000)]
0x0000000aa61c7800 JavaThread "WRITE-/172.16.1.12" daemon [_thread_blocked, id=-1507985152, stack(0x00007ffff151a000,0x00007ffff161a000)]
0x0000000aa61c7c00 JavaThread "WRITE-/172.16.1.12" daemon [_thread_blocked, id=-1507984704, stack(0x00007ffff161b000,0x00007ffff171b000)]
0x000000099452c000 JavaThread "HintedHandoff:1" daemon [_thread_blocked, id=-1806747520, stack(0x00007ffff191e000,0x00007ffff1a1e000)]
0x000000099452c800 JavaThread "ACCEPT-silo-16-1-11.cassandra.ebureau.com/172.16.1.11" [_thread_in_native, id=-1806747072, stack(0x00007ffff1a1f000,0x00007ffff1b1f000)]
0x000000099452cc00 JavaThread "GossipTasks:1" daemon [_thread_blocked, id=-1806746624, stack(0x00007ffff1b20000,0x00007ffff1c20000)]
0x0000000994538000 JavaThread "MiscStage:1" daemon [_thread_blocked, id=-1806735424, stack(0x00007ffff353a000,0x00007ffff363a000)]
0x0000000994538800 JavaThread "MigrationStage:1" daemon [_thread_blocked, id=-1806734976, stack(0x00007ffff363b000,0x00007ffff373b000)]
0x0000000994538c00 JavaThread "AntiEntropyStage:1" daemon [_thread_blocked, id=-1806734528, stack(0x00007ffff373c000,0x00007ffff383c000)]
0x0000000994539400 JavaThread "GossipStage:1" daemon [_thread_blocked, id=-1806734080, stack(0x00007ffff383d000,0x00007ffff393d000)]
0x0000000994539c00 JavaThread "StreamStage:1" daemon [_thread_blocked, id=-1806733632, stack(0x00007ffff393e000,0x00007ffff3a3e000)]
0x0000000801a8a400 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, id=27570624, stack(0x00007ffffc1c6000,0x00007ffffc2c6000)]
0x0000000801a8ac00 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, id=27571520, stack(0x00007ffffccd1000,0x00007ffffcdd1000)]
0x0000000801a8b400 JavaThread "COMMIT-LOG-ALLOCATOR" [_thread_blocked, id=27571968, stack(0x00007ffffc2c7000,0x00007ffffc3c7000)]
0x0000000801a8bc00 JavaThread "commitlog_archiver:1" daemon [_thread_blocked, id=-1927292224, stack(0x00007ffffcfd4000,0x00007ffffd0d4000)]
0x0000000801a98800 JavaThread "MemtablePostFlusher:1" daemon [_thread_blocked, id=27576000, stack(0x00007ffffd0d5000,0x00007ffffd1d5000)]
0x0000000801a99800 JavaThread "OptionalTasks:1" daemon [_thread_blocked, id=27576896, stack(0x00007ffffd2d7000,0x00007ffffd3d7000)]
0x0000000801a99c00 JavaThread "EXPIRING-MAP-REAPER:1" daemon [_thread_blocked, id=27577792, stack(0x00007ffffd3d8000,0x00007ffffd4d8000)]
0x0000000801a9a400 JavaThread "ScheduledTasks:1" daemon [_thread_blocked, id=27577344, stack(0x00007ffffd4d9000,0x00007ffffd5d9000)]
0x0000000801a9ac00 JavaThread "FileWatchdog" daemon [_thread_blocked, id=27578240, stack(0x00007ffffd5da000,0x00007ffffd6da000)]
0x0000000801a9b400 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=27578688, stack(0x00007ffffd7dc000,0x00007ffffd8dc000)]
0x0000000801a9b800 JavaThread "RMI TCP Accept-7199" daemon [_thread_in_native, id=27580928, stack(0x00007ffffd8dd000,0x00007ffffd9dd000)]
0x0000000801a9c000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=27581376, stack(0x00007ffffd9de000,0x00007ffffdade000)]
0x0000000801a9c800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=27581824, stack(0x00007ffffdadf000,0x00007ffffdbdf000)]
0x0000000801a9d000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=27582272, stack(0x00007ffffdbe0000,0x00007ffffdce0000)]
0x0000000801a9d400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=27582720, stack(0x00007ffffdce1000,0x00007ffffdde1000)]
0x0000000801a9fc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=27583616, stack(0x00007ffffdde2000,0x00007ffffdee2000)]
0x0000000801ae3000 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=27584064, stack(0x00007ffffdee3000,0x00007ffffdfe3000)]
0x0000000801aa0400 JavaThread "Finalizer" daemon [_thread_blocked, id=27584512, stack(0x00007ffffdfe4000,0x00007ffffe0e4000)]
0x0000000801aa0800 JavaThread "Reference Handler" daemon [_thread_blocked, id=27584960, stack(0x00007ffffe0e5000,0x00007ffffe1e5000)]
Other Threads:
0x0000000801af8c00 VMThread [stack: 0x00007ffffe1e6000,0x00007ffffe2e6000] [id=27585408]
0x0000000801af7800 WatcherThread [stack: 0x00007ffffd6db000,0x00007ffffd7db000] [id=27588992]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generation total 1413760K, used 264106K [0x0000000806800000, 0x0000000866600000, 0x0000000866600000)
eden space 1256704K, 20% used [0x0000000806800000, 0x00000008162da288, 0x0000000853340000)
from space 157056K, 4% used [0x000000085cca0000, 0x000000085d3b05b0, 0x0000000866600000)
to space 157056K, 0% used [0x0000000853340000, 0x0000000853340000, 0x000000085cca0000)
concurrent mark-sweep generation total 4713472K, used 1236338K [0x0000000866600000, 0x0000000986100000, 0x0000000986200000)
concurrent-mark-sweep perm gen total 41668K, used 25000K [0x0000000986200000, 0x0000000988ab1000, 0x000000098b600000)
Dynamic libraries:
0x0000000000400000 /usr/local/diablo-jdk1.6.0/bin/java
0x0000000800658000 /lib/libthr.so.3
0x0000000800771000 /lib/libc.so.7
0x0000000800c00000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/server/libjvm.so
0x00000008014a4000 /usr/lib/libstdc++.so.6
0x00000008016b4000 /lib/libm.so.5
0x00000008017d4000 /lib/libgcc_s.so.1
0x00000008018e2000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/native_threads/libhpi.so
0x0000000801c00000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libinstrument.so
0x0000000801d18000 /usr/local/lib/libiconv.so.3
0x0000000801f13000 /usr/local/lib/nss_ldap.so.1
0x0000000802035000 /usr/local/lib/libldap-2.4.so.8
0x0000000802176000 /usr/local/lib/liblber-2.4.so.8
0x0000000802283000 /usr/lib/libkrb5.so.10
0x00000008023f2000 /usr/lib/libcom_err.so.5
0x00000008024f4000 /usr/lib/libgssapi_krb5.so.10
0x000000080260e000 /usr/lib/libssl.so.6
0x0000000802761000 /lib/libcrypto.so.6
0x0000000802a01000 /usr/lib/libgssapi.so.10
0x0000000802b0b000 /usr/lib/libhx509.so.10
0x0000000802c4b000 /usr/lib/libroken.so.10
0x0000000802d5c000 /usr/lib/libasn1.so.10
0x0000000802ede000 /lib/libcrypt.so.5
0x0000000802ff7000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libverify.so
0x0000000803108000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libjava.so
0x000000080323c000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libzip.so
0x0000000803345000 /usr/local/lib/compat/libz.so.4
0x0000000993600000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libmanagement.so
0x0000000993708000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libnet.so
0x000000099381f000 /var/tmp/jna56081.tmp
0x0000000994a00000 /var/tmp/snappy-1.0.5-libsnappyjava.so
0x0000000994b07000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libnio.so
0x0000000994c0f000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/librmi.so
0x0000000800511000 /libexec/ld-elf.so.1
VM Arguments:
jvm_args: -ea -javaagent:/usr/local/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6137M -Xmx6137M -Xmn1534M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
java_command: org.apache.cassandra.thrift.CassandraDaemon
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/diablo-jdk1.6.0
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/spy/bin
LD_LIBRARY_PATH=/usr/local/diablo-jdk1.6.0/jre/lib/amd64/server:/usr/local/diablo-jdk1.6.0/jre/lib/amd64:/usr/local/diablo-jdk1.6.0/jre/../lib/amd64:/usr/lib:/usr/local/lib:/usr/local/pgsql/lib
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x617aa0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGBUS: [libjvm.so+0x617aa0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGFPE: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGPIPE: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGXFSZ: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGILL: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGUSR1: SIG_DFL, sa_mask[0]=0x63807efb, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x4e3fa0], sa_mask[0]=0x00000000, sa_flags=0x00000042
SIGHUP: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
SIGQUIT: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
--------------- S Y S T E M ---------------
OS:Bsd
uname:FreeBSD 8.2-STABLE FreeBSD 8.2-STABLE #0: Mon Feb 27 11:43:12 CST 2012 root@localhost:/usr/obj/usr/src/sys/GENERICDTRACE amd64
rlimit: STACK 524288k, CORE infinity, NPROC 5547, NOFILE 405000
CPU:total 24 (16 cores per cpu, 2 threads per core) family 6 model 12 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, ht
Memory: 4k page, physical 11034040k(2758510k free)
vm_info: Diablo Java HotSpot(TM) 64-Bit Server VM (10.0-b23) for bsd-amd64 JRE (1.6.0_07-b02), built on May 12 2008 20:56:50 by "truk" with gcc 4.2.1 20070719 [FreeBSD]
time: Wed Jun 6 17:54:20 2012
elapsed time: 2577 seconds
Re: Cassandra 1.1.1 crash during compaction
Posted by Sylvain Lebresne <sy...@datastax.com>.
A JVM crash means that it's unlikely linked to Cassandra itself. We do
use a few unsafe calls here and there, but nothing as changed on that
front for 1.1.1 and we never had any report of this crashing the JVM.
So imo there is a very good chance that this is a JVM bug. In
particular, I doubt many people have tested the Diablo JDK with
Cassandra and I would suggest preferring the sun JDK (the most tested)
or openJDK. I suppose there is also the possibility that snappy has a
bug on FreeBSD.
--
Sylvain
On Thu, Jun 7, 2012 at 7:51 PM, Dustin Wenz <du...@ebureau.com> wrote:
> We observed a JRE crash on one node in a seven node cluster about a half hour after upgrading to version 1.1.1 yesterday. Immediately after the upgrade, everything seemed to be working fine. The last item in the cassandra log was a info-level notification that compaction had started on a data file. Four minutes later, the process crashed.
>
> The host OS is FreeBSD 8.2, built for the amd64 architecture. Most of the cluster settings are left to their defaults and the replication factor is set to 2 for our keyspace. We are using the RandomPartitioner and RackInferringSnitch. JNA is enabled, but cannot use mlockall since the process runs as a non-privileged user. It was also necessary to build our own Snappy compressor jar file, since the required architecture was not built-in to the public distribution.
>
> Cassandra is a fairly new software deployment for us, and I was hoping someone could give me some pointers on interpreting the crash report below.
>
> Thanks,
>
> - .Dustin
>
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> # SIGBUS (0xa) at pc=0x0000000801199140, pid=44897, tid=0x8d1fdc80
> #
> # Java VM: Diablo Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode bsd-amd64)
> # Problematic frame:
> # V [libjvm.so+0x599140]
> #
> # Please submit bug reports to freebsd-java@FreeBSD.org
> #
>
> --------------- T H R E A D ---------------
>
> Current thread (0x0000000aa9d56000): JavaThread "CompactionExecutor:30" daemon [_thread_in_vm, id=-1927291776, stack(0x00007ffff898e000,0x00007ffff8a8e000)]
>
> siginfo:si_signo=SIGBUS: si_errno=0, si_code=3 (BUS_OBJERR), si_addr=0x0000000801199140
>
> Registers:
> RAX=0x0000000aa95f8fe8, RBX=0x0000000aa95fc2c0, RCX=0x0000000987a334f0, RDX=0x00000009274e9888
> RSP=0x00007ffff8a8d630, RBP=0x00007ffff8a8d640, RSI=0x00000009274e9888, RDI=0xc900000009274e99
> R8 =0x000000098c203bd8, R9 =0x00000008809ff4b8, R10=0x0000000801488580, R11=0x0000000000000001
> R12=0x0000000aa9d56000, R13=0x0000000aa95f8c00, R14=0x00007ffff8a8d818, R15=0x0000000aa95f8c10
> RIP=0x0000000801199140, EFL=0x003b003b00000001, ERR=0x0000000000000000
> TRAPNO=0x001b001300000009
>
> Top of Stack: (sp=0x00007ffff8a8d630)
> 0x00007ffff8a8d630: 0000000aa95fc2c0 0000000aa9d56000
> 0x00007ffff8a8d640: 00007ffff8a8d660 000000080119917e
> 0x00007ffff8a8d650: 000000080345b690 0000000aa95fc2c0
> 0x00007ffff8a8d660: 00007ffff8a8d6a0 0000000800f3a1fd
> 0x00007ffff8a8d670: 0000000aa95f8fe8 00000009274e9869
> 0x00007ffff8a8d680: 0000000986f7acb0 0000000986f7e591
> 0x00007ffff8a8d690: 00007ffff8a8d818 0000000aa9d56000
> 0x00007ffff8a8d6a0: 00007ffff8a8d700 000000080346556f
> 0x00007ffff8a8d6b0: 00000009274e9888 000000080346553b
> 0x00007ffff8a8d6c0: 00007ffff8a8d6c0 0000000986f7e591
> 0x00007ffff8a8d6d0: 00007ffff8a8d818 0000000987a30198
> 0x00007ffff8a8d6e0: 0000000987a334f0 0000000986f7e918
> 0x00007ffff8a8d6f0: 0000000000000000 00007ffff8a8d810
> 0x00007ffff8a8d700: 00007ffff8a8d870 000000080345c04e
> 0x00007ffff8a8d710: 0000000000000000 0000000000000000
> 0x00007ffff8a8d720: 0000000000000000 0000000000000000
> 0x00007ffff8a8d730: 0000000000000000 0000000000000000
> 0x00007ffff8a8d740: 00000009274aa428 000000080817df80
> 0x00007ffff8a8d750: 00000008081714c0 0000000000000000
> 0x00007ffff8a8d760: 000000080817df30 0000000000000000
> 0x00007ffff8a8d770: 000000091b5ed0a0 0000000000000000
> 0x00007ffff8a8d780: 0000000000000000 0000000000000000
> 0x00007ffff8a8d790: 0000000000000000 0000000000000000
> 0x00007ffff8a8d7a0: 0000000000000000 0000000000000000
> 0x00007ffff8a8d7b0: 0000000000000000 0000000000000000
> 0x00007ffff8a8d7c0: 0000000000000000 0000000000000000
> 0x00007ffff8a8d7d0: 0000000000000000 000000000017974b
> 0x00007ffff8a8d7e0: deaddeaf00000000 00000137c3fdd558
> 0x00007ffff8a8d7f0: deaddeaf00000000 0000000000000000
> 0x00007ffff8a8d800: 0000000000000000 000000091b638910
> 0x00007ffff8a8d810: 0000000000000000 0000000000000000
> 0x00007ffff8a8d820: 0000000882d0d2e8 00000008406a3de8
>
> Instructions: (pc=0x0000000801199140)
> 0x0000000801199130: 55 48 89 e5 48 89 5d f0 4c 89 65 f8 48 83 ec 10
> 0x0000000801199140: 0f b7 47 10 48 89 fb 44 8d 60 01 49 63 fc e8 4d
>
> Stack: [0x00007ffff898e000,0x00007ffff8a8e000], sp=0x00007ffff8a8d630, free space=1021k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V [libjvm.so+0x599140]
> V [libjvm.so+0x59917e]
> V [libjvm.so+0x33a1fd]
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> v ~BufferBlob::Interpreter
> J java.util.concurrent.ThreadPoolExecutor$Worker.run()V
> v ~BufferBlob::Interpreter
> v ~BufferBlob::StubRoutines (1)
>
> --------------- P R O C E S S ---------------
>
> Java Threads: ( => current thread )
> =>0x0000000aa9d56000 JavaThread "CompactionExecutor:30" daemon [_thread_in_vm, id=-1927291776, stack(0x00007ffff898e000,0x00007ffff8a8e000)]
> 0x00000009962e8000 JavaThread "CompactionExecutor:29" daemon [_thread_in_native, id=-1927291328, stack(0x00007ffff787d000,0x00007ffff797d000)]
> 0x0000000aa6070400 JavaThread "Thread-58" [_thread_blocked, id=-1823155584, stack(0x00007ffffc7cc000,0x00007ffffc8cc000)]
> 0x0000000aa7f1c400 JavaThread "Streaming to /172.16.1.17:1" daemon [_thread_in_native, id=-1509346816, stack(0x00007ffffbfc4000,0x00007ffffc0c4000)]
> 0x0000000aa7c37000 JavaThread "NonPeriodicTasks:1" daemon [_thread_blocked, id=-1927303872, stack(0x00007ffff8085000,0x00007ffff8185000)]
> 0x0000000aa9bf2000 JavaThread "MemoryMeter:1" daemon [_thread_blocked, id=-1823136768, stack(0x00007ffffcacf000,0x00007ffffcbcf000)]
> 0x0000000aa95c8c00 JavaThread "CompactionExecutor:24" daemon [_thread_in_Java, id=-1806730048, stack(0x00007ffff757a000,0x00007ffff767a000)]
> 0x0000000aa6074000 JavaThread "Thread-50" [_thread_blocked, id=-1823156032, stack(0x00007ffffb0b5000,0x00007ffffb1b5000)]
> 0x0000000aa6077800 JavaThread "Thread-42" [_thread_in_native, id=-1927310144, stack(0x00007ffff868b000,0x00007ffff878b000)]
> 0x0000000aa9422000 JavaThread "AntiEntropySessions:5" daemon [_thread_blocked, id=-1927308352, stack(0x00007ffffadb2000,0x00007ffffaeb2000)]
> 0x0000000aa7f1cc00 JavaThread "Streaming to /172.16.1.12:1" daemon [_thread_in_native, id=-1927309248, stack(0x00007ffffcdd2000,0x00007ffffced2000)]
> 0x0000000aa69e0400 JavaThread "Thrift:16" daemon [_thread_blocked, id=-1806731840, stack(0x00007ffffbdc2000,0x00007ffffbec2000)]
> 0x0000000aa69e0800 JavaThread "Thrift:15" daemon [_thread_blocked, id=-1806729152, stack(0x00007ffff767b000,0x00007ffff777b000)]
> 0x0000000aa6078400 JavaThread "Thread-39" [_thread_in_native, id=-1806731392, stack(0x00007ffff777c000,0x00007ffff787c000)]
> 0x0000000aa9423c00 JavaThread "AntiEntropySessions:1" daemon [_thread_blocked, id=-1823156928, stack(0x00007ffffc5ca000,0x00007ffffc6ca000)]
> 0x0000000aa9424400 JavaThread "JMX server connection timeout 432" daemon [_thread_blocked, id=-1509345920, stack(0x00007ffff7c81000,0x00007ffff7d81000)]
> 0x0000000aa69e1000 JavaThread "Thrift:14" daemon [_thread_blocked, id=-1823155136, stack(0x00007ffffaeb3000,0x00007ffffafb3000)]
> 0x0000000994701400 JavaThread "RMI TCP Connection(34)-172.16.1.11" daemon [_thread_blocked, id=-1823140800, stack(0x00007ffff858a000,0x00007ffff868a000)]
> 0x00000009949b2c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=-1823152000, stack(0x00007ffff8489000,0x00007ffff8589000)]
> 0x0000000aa69e1800 JavaThread "Thrift:13" daemon [_thread_blocked, id=-1823151104, stack(0x00007ffff7a7f000,0x00007ffff7b7f000)]
> 0x0000000aa6078c00 JavaThread "Thread-35" [_thread_in_native, id=-1823133632, stack(0x00007ffff878c000,0x00007ffff888c000)]
> 0x0000000aa69e2000 JavaThread "Thrift:12" daemon [_thread_blocked, id=-1806733184, stack(0x00007ffffacb1000,0x00007ffffadb1000)]
> 0x0000000aa6079800 JavaThread "Thread-30" [_thread_in_native, id=-1806732736, stack(0x00007ffff8388000,0x00007ffff8488000)]
> 0x0000000aa69e2400 JavaThread "Thrift:11" daemon [_thread_blocked, id=-1823157824, stack(0x00007ffff8287000,0x00007ffff8387000)]
> 0x0000000aa69e2c00 JavaThread "Thrift:10" daemon [_thread_in_native, id=-1823153792, stack(0x00007ffffabb0000,0x00007ffffacb0000)]
> 0x0000000aa69e3400 JavaThread "Thrift:9" daemon [_thread_blocked, id=-1823153344, stack(0x00007ffffaaaf000,0x00007ffffabaf000)]
> 0x0000000aa69e3c00 JavaThread "Thrift:8" daemon [_thread_blocked, id=-1823141248, stack(0x00007ffff090e000,0x00007ffff0a0e000)]
> 0x0000000aa69e4000 JavaThread "Thrift:7" daemon [_thread_blocked, id=-1927307904, stack(0x00007ffffbcc1000,0x00007ffffbdc1000)]
> 0x0000000aa607a800 JavaThread "Thread-22" [_thread_in_native, id=-1927307456, stack(0x00007ffffb8bd000,0x00007ffffb9bd000)]
> 0x0000000aa69e4800 JavaThread "Thrift:6" daemon [_thread_blocked, id=-1927306560, stack(0x00007ffffb5ba000,0x00007ffffb6ba000)]
> 0x0000000aa69e5000 JavaThread "Thrift:5" daemon [_thread_blocked, id=-1927305664, stack(0x00007ffffb3b8000,0x00007ffffb4b8000)]
> 0x0000000aa69e5800 JavaThread "Thrift:4" daemon [_thread_blocked, id=-1927304320, stack(0x00007ffffcbd0000,0x00007ffffccd0000)]
> 0x0000000aa69e5c00 JavaThread "Thrift:3" daemon [_thread_blocked, id=-1498747776, stack(0x00007ffffb7bc000,0x00007ffffb8bc000)]
> 0x0000000aa69e6400 JavaThread "Thrift:2" daemon [_thread_blocked, id=-1509348160, stack(0x00007ffff0b10000,0x00007ffff0c10000)]
> 0x0000000aa69e6c00 JavaThread "Thrift:1" daemon [_thread_blocked, id=-1509347264, stack(0x00007ffff080d000,0x00007ffff090d000)]
> 0x0000000801aa2000 JavaThread "DestroyJavaVM" [_thread_blocked, id=10530368, stack(0x00007fffffaff000,0x00007fffffbff000)]
> 0x000000099452b000 JavaThread "Thread-16" [_thread_in_native, id=-1806747968, stack(0x00007fffeff04000,0x00007ffff0004000)]
> 0x000000099452b800 JavaThread "Thread-15" [_thread_in_native, id=-1806748416, stack(0x00007ffff0005000,0x00007ffff0105000)]
> 0x0000000aa607b400 JavaThread "Thread-13" [_thread_in_native, id=-1509348608, stack(0x00007ffff0409000,0x00007ffff0509000)]
> 0x00000009bdcb5400 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=-1115419520, stack(0x00007ffff060b000,0x00007ffff070b000)]
> 0x0000000aa607c400 JavaThread "Thread-11" [_thread_in_native, id=-1509347712, stack(0x00007ffff070c000,0x00007ffff080c000)]
> 0x0000000aa61c3000 JavaThread "WRITE-/172.16.1.14" daemon [_thread_blocked, id=-1507987840, stack(0x00007ffff0c11000,0x00007ffff0d11000)]
> 0x0000000aa61c3800 JavaThread "WRITE-/172.16.1.14" daemon [_thread_blocked, id=-1507987392, stack(0x00007ffff0d12000,0x00007ffff0e12000)]
> 0x0000000aa61c4000 JavaThread "WRITE-/172.16.1.13" daemon [_thread_blocked, id=-1509345024, stack(0x00007ffff1419000,0x00007ffff1519000)]
> 0x0000000aa61c4400 JavaThread "WRITE-/172.16.1.13" daemon [_thread_blocked, id=-1509344576, stack(0x00007ffff171c000,0x00007ffff181c000)]
> 0x0000000aa61c4c00 JavaThread "WRITE-/172.16.1.16" daemon [_thread_blocked, id=-1509344128, stack(0x00007ffff181d000,0x00007ffff191d000)]
> 0x0000000aa61c5400 JavaThread "WRITE-/172.16.1.16" daemon [_thread_blocked, id=27571072, stack(0x00007ffffced3000,0x00007ffffcfd3000)]
> 0x0000000aa607f000 JavaThread "Thread-5" [_thread_in_native, id=-1509345472, stack(0x00007ffff0f14000,0x00007ffff1014000)]
> 0x0000000aa61c5c00 JavaThread "WRITE-/172.16.1.17" daemon [_thread_blocked, id=-1507986944, stack(0x00007ffff1015000,0x00007ffff1115000)]
> 0x0000000aa61c6000 JavaThread "WRITE-/172.16.1.17" daemon [_thread_blocked, id=-1507986496, stack(0x00007ffff1116000,0x00007ffff1216000)]
> 0x0000000aa61c6800 JavaThread "WRITE-/172.16.1.15" daemon [_thread_blocked, id=-1507986048, stack(0x00007ffff1217000,0x00007ffff1317000)]
> 0x0000000aa61c7000 JavaThread "WRITE-/172.16.1.15" daemon [_thread_blocked, id=-1507985600, stack(0x00007ffff1318000,0x00007ffff1418000)]
> 0x0000000aa61c7800 JavaThread "WRITE-/172.16.1.12" daemon [_thread_blocked, id=-1507985152, stack(0x00007ffff151a000,0x00007ffff161a000)]
> 0x0000000aa61c7c00 JavaThread "WRITE-/172.16.1.12" daemon [_thread_blocked, id=-1507984704, stack(0x00007ffff161b000,0x00007ffff171b000)]
> 0x000000099452c000 JavaThread "HintedHandoff:1" daemon [_thread_blocked, id=-1806747520, stack(0x00007ffff191e000,0x00007ffff1a1e000)]
> 0x000000099452c800 JavaThread "ACCEPT-silo-16-1-11.cassandra.ebureau.com/172.16.1.11" [_thread_in_native, id=-1806747072, stack(0x00007ffff1a1f000,0x00007ffff1b1f000)]
> 0x000000099452cc00 JavaThread "GossipTasks:1" daemon [_thread_blocked, id=-1806746624, stack(0x00007ffff1b20000,0x00007ffff1c20000)]
> 0x0000000994538000 JavaThread "MiscStage:1" daemon [_thread_blocked, id=-1806735424, stack(0x00007ffff353a000,0x00007ffff363a000)]
> 0x0000000994538800 JavaThread "MigrationStage:1" daemon [_thread_blocked, id=-1806734976, stack(0x00007ffff363b000,0x00007ffff373b000)]
> 0x0000000994538c00 JavaThread "AntiEntropyStage:1" daemon [_thread_blocked, id=-1806734528, stack(0x00007ffff373c000,0x00007ffff383c000)]
> 0x0000000994539400 JavaThread "GossipStage:1" daemon [_thread_blocked, id=-1806734080, stack(0x00007ffff383d000,0x00007ffff393d000)]
> 0x0000000994539c00 JavaThread "StreamStage:1" daemon [_thread_blocked, id=-1806733632, stack(0x00007ffff393e000,0x00007ffff3a3e000)]
> 0x0000000801a8a400 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, id=27570624, stack(0x00007ffffc1c6000,0x00007ffffc2c6000)]
> 0x0000000801a8ac00 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, id=27571520, stack(0x00007ffffccd1000,0x00007ffffcdd1000)]
> 0x0000000801a8b400 JavaThread "COMMIT-LOG-ALLOCATOR" [_thread_blocked, id=27571968, stack(0x00007ffffc2c7000,0x00007ffffc3c7000)]
> 0x0000000801a8bc00 JavaThread "commitlog_archiver:1" daemon [_thread_blocked, id=-1927292224, stack(0x00007ffffcfd4000,0x00007ffffd0d4000)]
> 0x0000000801a98800 JavaThread "MemtablePostFlusher:1" daemon [_thread_blocked, id=27576000, stack(0x00007ffffd0d5000,0x00007ffffd1d5000)]
> 0x0000000801a99800 JavaThread "OptionalTasks:1" daemon [_thread_blocked, id=27576896, stack(0x00007ffffd2d7000,0x00007ffffd3d7000)]
> 0x0000000801a99c00 JavaThread "EXPIRING-MAP-REAPER:1" daemon [_thread_blocked, id=27577792, stack(0x00007ffffd3d8000,0x00007ffffd4d8000)]
> 0x0000000801a9a400 JavaThread "ScheduledTasks:1" daemon [_thread_blocked, id=27577344, stack(0x00007ffffd4d9000,0x00007ffffd5d9000)]
> 0x0000000801a9ac00 JavaThread "FileWatchdog" daemon [_thread_blocked, id=27578240, stack(0x00007ffffd5da000,0x00007ffffd6da000)]
> 0x0000000801a9b400 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=27578688, stack(0x00007ffffd7dc000,0x00007ffffd8dc000)]
> 0x0000000801a9b800 JavaThread "RMI TCP Accept-7199" daemon [_thread_in_native, id=27580928, stack(0x00007ffffd8dd000,0x00007ffffd9dd000)]
> 0x0000000801a9c000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=27581376, stack(0x00007ffffd9de000,0x00007ffffdade000)]
> 0x0000000801a9c800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=27581824, stack(0x00007ffffdadf000,0x00007ffffdbdf000)]
> 0x0000000801a9d000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=27582272, stack(0x00007ffffdbe0000,0x00007ffffdce0000)]
> 0x0000000801a9d400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=27582720, stack(0x00007ffffdce1000,0x00007ffffdde1000)]
> 0x0000000801a9fc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=27583616, stack(0x00007ffffdde2000,0x00007ffffdee2000)]
> 0x0000000801ae3000 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=27584064, stack(0x00007ffffdee3000,0x00007ffffdfe3000)]
> 0x0000000801aa0400 JavaThread "Finalizer" daemon [_thread_blocked, id=27584512, stack(0x00007ffffdfe4000,0x00007ffffe0e4000)]
> 0x0000000801aa0800 JavaThread "Reference Handler" daemon [_thread_blocked, id=27584960, stack(0x00007ffffe0e5000,0x00007ffffe1e5000)]
>
> Other Threads:
> 0x0000000801af8c00 VMThread [stack: 0x00007ffffe1e6000,0x00007ffffe2e6000] [id=27585408]
> 0x0000000801af7800 WatcherThread [stack: 0x00007ffffd6db000,0x00007ffffd7db000] [id=27588992]
>
> VM state:not at safepoint (normal execution)
>
> VM Mutex/Monitor currently owned by a thread: None
>
> Heap
> par new generation total 1413760K, used 264106K [0x0000000806800000, 0x0000000866600000, 0x0000000866600000)
> eden space 1256704K, 20% used [0x0000000806800000, 0x00000008162da288, 0x0000000853340000)
> from space 157056K, 4% used [0x000000085cca0000, 0x000000085d3b05b0, 0x0000000866600000)
> to space 157056K, 0% used [0x0000000853340000, 0x0000000853340000, 0x000000085cca0000)
> concurrent mark-sweep generation total 4713472K, used 1236338K [0x0000000866600000, 0x0000000986100000, 0x0000000986200000)
> concurrent-mark-sweep perm gen total 41668K, used 25000K [0x0000000986200000, 0x0000000988ab1000, 0x000000098b600000)
>
> Dynamic libraries:
> 0x0000000000400000 /usr/local/diablo-jdk1.6.0/bin/java
> 0x0000000800658000 /lib/libthr.so.3
> 0x0000000800771000 /lib/libc.so.7
> 0x0000000800c00000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/server/libjvm.so
> 0x00000008014a4000 /usr/lib/libstdc++.so.6
> 0x00000008016b4000 /lib/libm.so.5
> 0x00000008017d4000 /lib/libgcc_s.so.1
> 0x00000008018e2000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/native_threads/libhpi.so
> 0x0000000801c00000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libinstrument.so
> 0x0000000801d18000 /usr/local/lib/libiconv.so.3
> 0x0000000801f13000 /usr/local/lib/nss_ldap.so.1
> 0x0000000802035000 /usr/local/lib/libldap-2.4.so.8
> 0x0000000802176000 /usr/local/lib/liblber-2.4.so.8
> 0x0000000802283000 /usr/lib/libkrb5.so.10
> 0x00000008023f2000 /usr/lib/libcom_err.so.5
> 0x00000008024f4000 /usr/lib/libgssapi_krb5.so.10
> 0x000000080260e000 /usr/lib/libssl.so.6
> 0x0000000802761000 /lib/libcrypto.so.6
> 0x0000000802a01000 /usr/lib/libgssapi.so.10
> 0x0000000802b0b000 /usr/lib/libhx509.so.10
> 0x0000000802c4b000 /usr/lib/libroken.so.10
> 0x0000000802d5c000 /usr/lib/libasn1.so.10
> 0x0000000802ede000 /lib/libcrypt.so.5
> 0x0000000802ff7000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libverify.so
> 0x0000000803108000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libjava.so
> 0x000000080323c000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libzip.so
> 0x0000000803345000 /usr/local/lib/compat/libz.so.4
> 0x0000000993600000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libmanagement.so
> 0x0000000993708000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libnet.so
> 0x000000099381f000 /var/tmp/jna56081.tmp
> 0x0000000994a00000 /var/tmp/snappy-1.0.5-libsnappyjava.so
> 0x0000000994b07000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/libnio.so
> 0x0000000994c0f000 /usr/local/diablo-jdk1.6.0/jre/lib/amd64/librmi.so
> 0x0000000800511000 /libexec/ld-elf.so.1
>
> VM Arguments:
> jvm_args: -ea -javaagent:/usr/local/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6137M -Xmx6137M -Xmn1534M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
> java_command: org.apache.cassandra.thrift.CassandraDaemon
> Launcher Type: SUN_STANDARD
>
> Environment Variables:
> JAVA_HOME=/usr/local/diablo-jdk1.6.0
> PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/spy/bin
> LD_LIBRARY_PATH=/usr/local/diablo-jdk1.6.0/jre/lib/amd64/server:/usr/local/diablo-jdk1.6.0/jre/lib/amd64:/usr/local/diablo-jdk1.6.0/jre/../lib/amd64:/usr/lib:/usr/local/lib:/usr/local/pgsql/lib
> SHELL=/bin/bash
>
> Signal Handlers:
> SIGSEGV: [libjvm.so+0x617aa0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGBUS: [libjvm.so+0x617aa0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGFPE: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGPIPE: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGXFSZ: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGILL: [libjvm.so+0x4e17a0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGUSR1: SIG_DFL, sa_mask[0]=0x63807efb, sa_flags=0x00000000
> SIGUSR2: [libjvm.so+0x4e3fa0], sa_mask[0]=0x00000000, sa_flags=0x00000042
> SIGHUP: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGTERM: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
> SIGQUIT: [libjvm.so+0x4e2ca0], sa_mask[0]=0xfffefeff, sa_flags=0x00000042
>
>
> --------------- S Y S T E M ---------------
>
> OS:Bsd
> uname:FreeBSD 8.2-STABLE FreeBSD 8.2-STABLE #0: Mon Feb 27 11:43:12 CST 2012 root@localhost:/usr/obj/usr/src/sys/GENERICDTRACE amd64
> rlimit: STACK 524288k, CORE infinity, NPROC 5547, NOFILE 405000
> CPU:total 24 (16 cores per cpu, 2 threads per core) family 6 model 12 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, ht
>
> Memory: 4k page, physical 11034040k(2758510k free)
>
> vm_info: Diablo Java HotSpot(TM) 64-Bit Server VM (10.0-b23) for bsd-amd64 JRE (1.6.0_07-b02), built on May 12 2008 20:56:50 by "truk" with gcc 4.2.1 20070719 [FreeBSD]
>
> time: Wed Jun 6 17:54:20 2012
> elapsed time: 2577 seconds
>