You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/05/05 08:24:39 UTC
cassandra jvm crash in GCTaskThread
Running a cluster of 0.6.1, one of the hosts crashed during GC.
Can anyone suggest what needs tweaking to prevent that?
The last configuration change I made was to
change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I can't
determine if it's related.
System is 64 bit centos, the rest of the system info is at the bottom of the
email.
The last lines before the crash from system.log are:
INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913 CompactionManager.java
(line 246) Compacting
[org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line 407)
Discarding obsolete commit
log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820 CompactionManager.java
(line 326) Compacted to
/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
291072443/291071793 bytes for 123292 keys. Time: 19906ms.
And a summary of hs_err is here. The rest is attached. Let me know if you've
see this before and can advice, thanks!
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
#
# JRE version: 6.0_17-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x1ffb76]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x0000000047276800): GCTaskThread [stack:
0x0000000000000000,0x0000000000000000] [id=974]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
si_addr=0x0000000000000000
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1" [_thread_blocked,
id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
[_thread_blocked, id=1270, stack(0x0000000049dbf000,0x0000000049ec0000)]
0x0000000047691000 JavaThread "Thread-26" [_thread_in_native, id=1191,
stack(0x0000000043d53000,0x0000000043e54000)]
0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native, id=1117,
stack(0x0000000048daf000,0x0000000048eb0000)]
0x0000000047a42800 JavaThread "WRITE-/192.168.252.61" [_thread_blocked,
id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
0x0000000047abc800 JavaThread "WRITE-/192.168.252.62" [_thread_blocked,
id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked, id=1108,
stack(0x0000000049abc000,0x0000000049bbd000)]
0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked,
id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
0x0000000047693800 JavaThread "Thread-16" [_thread_in_native, id=1106,
stack(0x00000000498ba000,0x00000000499bb000)]
0x0000000047693000 JavaThread "WRITE-/192.168.252.99" [_thread_blocked,
id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
stack(0x00000000496b8000,0x00000000497b9000)]
0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native, id=1103,
stack(0x00000000495b7000,0x00000000496b8000)]
0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59" [_thread_blocked,
id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native, id=1101,
stack(0x00000000493b5000,0x00000000494b6000)]
0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61" [_thread_blocked,
id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58" [_thread_blocked,
id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62" [_thread_blocked,
id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
0x00000000475db000 JavaThread "Thread-12" [_thread_in_native, id=1097,
stack(0x0000000048fb1000,0x00000000490b2000)]
0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
stack(0x0000000048eb0000,0x0000000048fb1000)]
0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
stack(0x0000000048bad000,0x0000000048cae000)]
0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
stack(0x0000000048aac000,0x0000000048bad000)]
0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked, id=1091,
stack(0x00000000489ab000,0x0000000048aac000)]
0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked, id=1090,
stack(0x00000000488aa000,0x00000000489ab000)]
0x00002aabbc1ac000 JavaThread "
ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked, id=1088,
stack(0x0000000040694000,0x0000000040795000)]
0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
[_thread_blocked, id=1086, stack(0x00000000485a7000,0x00000000486a8000)]
0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
id=1085, stack(0x0000000040830000,0x0000000040931000)]
0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked,
id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
stack(0x00000000482a4000,0x00000000483a5000)]
0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked, id=1078,
stack(0x00000000481a3000,0x00000000482a4000)]
0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
id=1071, stack(0x0000000047086000,0x0000000047187000)]
0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
id=1070, stack(0x0000000046f85000,0x0000000047086000)]
0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
id=1064, stack(0x000000004697f000,0x0000000046a80000)]
0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
id=1063, stack(0x000000004687e000,0x000000004697f000)]
0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
id=1062, stack(0x000000004677d000,0x000000004687e000)]
0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
id=1061, stack(0x000000004667c000,0x000000004677d000)]
0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
id=1060, stack(0x000000004657b000,0x000000004667c000)]
0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
id=1059, stack(0x000000004647a000,0x000000004657b000)]
0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
id=1058, stack(0x0000000046379000,0x000000004647a000)]
0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
id=1057, stack(0x0000000046278000,0x0000000046379000)]
0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
id=1056, stack(0x0000000046177000,0x0000000046278000)]
0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
id=1055, stack(0x0000000046076000,0x0000000046177000)]
0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
id=1054, stack(0x0000000045f75000,0x0000000046076000)]
0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked,
id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked,
id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked,
id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked,
id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked,
id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked,
id=1048, stack(0x000000004596f000,0x0000000045a70000)]
0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked,
id=1047, stack(0x000000004586e000,0x000000004596f000)]
0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked,
id=1046, stack(0x000000004576d000,0x000000004586e000)]
0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked,
id=1045, stack(0x000000004566c000,0x000000004576d000)]
0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked,
id=1044, stack(0x000000004556b000,0x000000004566c000)]
0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked,
id=1043, stack(0x000000004546a000,0x000000004556b000)]
0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked,
id=1042, stack(0x0000000045369000,0x000000004546a000)]
0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked,
id=1041, stack(0x0000000045268000,0x0000000045369000)]
0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked,
id=1040, stack(0x0000000045167000,0x0000000045268000)]
0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked,
id=1039, stack(0x0000000045066000,0x0000000045167000)]
0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked,
id=1038, stack(0x0000000044f65000,0x0000000045066000)]
0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked,
id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked,
id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked,
id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked,
id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked,
id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked,
id=1032, stack(0x000000004495f000,0x0000000044a60000)]
0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked,
id=1031, stack(0x000000004485e000,0x000000004495f000)]
0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
id=1030, stack(0x000000004475d000,0x000000004485e000)]
0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
id=1029, stack(0x000000004465c000,0x000000004475d000)]
0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
id=1028, stack(0x000000004455b000,0x000000004465c000)]
0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
id=1027, stack(0x000000004445a000,0x000000004455b000)]
0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
id=1026, stack(0x0000000044359000,0x000000004445a000)]
0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
id=1025, stack(0x0000000044258000,0x0000000044359000)]
0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
id=1024, stack(0x0000000044157000,0x0000000044258000)]
0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
id=1023, stack(0x0000000044056000,0x0000000044157000)]
0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
id=1022, stack(0x0000000043f55000,0x0000000044056000)]
0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked,
id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked,
id=1017, stack(0x0000000041499000,0x000000004159a000)]
0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=1009,
stack(0x0000000043c52000,0x0000000043d53000)]
0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1" [_thread_blocked,
id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
[_thread_blocked, id=1007, stack(0x0000000043a50000,0x0000000043b51000)]
0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
[_thread_blocked, id=1006, stack(0x000000004394f000,0x0000000043a50000)]
0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
[_thread_blocked, id=1005, stack(0x000000004384e000,0x000000004394f000)]
0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked,
id=1004, stack(0x000000004374d000,0x000000004384e000)]
0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked,
id=1003, stack(0x000000004364c000,0x000000004374d000)]
0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked,
id=1002, stack(0x000000004354b000,0x000000004364c000)]
0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked,
id=1001, stack(0x000000004344a000,0x000000004354b000)]
0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1" [_thread_blocked,
id=1000, stack(0x0000000043349000,0x000000004344a000)]
0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
id=999, stack(0x0000000043248000,0x0000000043349000)]
0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
id=998, stack(0x0000000043147000,0x0000000043248000)]
0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked, id=997,
stack(0x000000004117f000,0x0000000041280000)]
0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
[_thread_in_native, id=995, stack(0x0000000040f7d000,0x000000004107e000)]
0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
[_thread_in_native, id=994, stack(0x0000000043046000,0x0000000043147000)]
0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
[_thread_in_native, id=989, stack(0x0000000040a48000,0x0000000040b49000)]
0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
[_thread_blocked, id=988, stack(0x0000000042f45000,0x0000000043046000)]
0x00002aabbc024000 JavaThread "CompilerThread1" daemon [_thread_blocked,
id=987, stack(0x0000000042e44000,0x0000000042f45000)]
0x00002aabbc022000 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=986, stack(0x0000000042d43000,0x0000000042e44000)]
0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
id=985, stack(0x00000000417e8000,0x00000000418e9000)]
0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
[_thread_blocked, id=984, stack(0x00000000416e7000,0x00000000417e8000)]
0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked, id=983,
stack(0x0000000042c42000,0x0000000042d43000)]
0x0000000047424800 JavaThread "Reference Handler" daemon [_thread_blocked,
id=982, stack(0x0000000042b41000,0x0000000042c42000)]
0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
stack(0x00000000412b1000,0x00000000413b2000)]
Other Threads:
0x000000004741d800 VMThread [stack: 0x0000000042a40000,0x0000000042b41000]
[id=981]
0x00002aabbc181000 WatcherThread [stack:
0x000000004107e000,0x000000004117f000] [id=996]
=>0x0000000047276800 (exited) GCTaskThread [stack:
0x0000000000000000,0x0000000000000000] [id=974]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
[0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
Heap
par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
0x00002aaabf050000, 0x00002aaabf050000)
eden space 272640K, 100% used [0x00002aaaae1f0000, 0x00002aaabec30000,
0x00002aaabec30000)
from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
0x00002aaabf050000)
to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
0x00002aaabee40000)
concurrent mark-sweep generation total 3184972K, used 2657590K
[0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
concurrent-mark-sweep perm gen total 26272K, used 15697K
[0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
VM Arguments:
jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90 -XX:+AggressiveOpts
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
-XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
-Dcassandra-pidfile=/var/run/cassandra.pid
java_command: org.apache.cassandra.thrift.CassandraDaemon
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/jdk1.6.0_17/
PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:CentOS release 5.4 (Final)
uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS infinity
load average:2.50 3.93 5.66
CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, ht
Memory: 4k page, physical 16429856k(55128k free), swap 5242872k(5220216k
free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64 JRE
(1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc 3.2.2
(SuSE Linux)
time: Tue May 4 17:03:44 2010
elapsed time: 11909 seconds
Re: cassandra jvm crash in GCTaskThread
Posted by Ran Tavory <ra...@gmail.com>.
I'll try that, thanks Nate
On Wed, May 5, 2010 at 10:00 AM, Nathan McCall <na...@vervewireless.com>wrote:
> Ran,
> You may want to upgrade your jvm. I had a similar core with a heavily
> loaded tomcat recently and found the following on the release notes
> page for 1.6.0_18:
> "Card-Marking Optimization Issue
>
> A flaw in the implementation of a card-marking performance
> optimization in the JVM can cause heap corruption under some
> circumstances. This issue affects the CMS garbage collector prior to
> 6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The
> serial garbage collector is not affected. Applications most likely to
> be affected by this issue are those that allocate very large objects
> which would not normally fit in Eden, or those that make extensive use
> of JNI Critical Sections (JNI Get/Release*Critical)."
>
> http://java.sun.com/javase/6/webnotes/6u18.html
>
> Note that there is an apparent work-around jvm option to add:
> -XX:-ReduceInitialCardMarks
>
> -Nate
>
>
> On Tue, May 4, 2010 at 11:24 PM, Ran Tavory <ra...@gmail.com> wrote:
> > Running a cluster of 0.6.1, one of the hosts crashed during GC.
> > Can anyone suggest what needs tweaking to prevent that?
> > The last configuration change I made was to
> > change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I
> can't
> > determine if it's related.
> >
> > System is 64 bit centos, the rest of the system info is at the bottom of
> the
> > email.
> > The last lines before the crash from system.log are:
> >
> > INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913 CompactionManager.java
> > (line 246) Compacting
> >
> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
> > INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line
> 407)
> > Discarding obsolete commit
> >
> log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
> > INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820 CompactionManager.java
> > (line 326) Compacted to
> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
> > 291072443/291071793 bytes for 123292 keys. Time: 19906ms.
> > And a summary of hs_err is here. The rest is attached. Let me know if
> you've
> > see this before and can advice, thanks!
> > #
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > # SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
> > #
> > # JRE version: 6.0_17-b04
> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
> > linux-amd64 )
> > # Problematic frame:
> > # V [libjvm.so+0x1ffb76]
> > #
> > # If you would like to submit a bug report, please visit:
> > # http://java.sun.com/webapps/bugreport/crash.jsp
> > #
> > --------------- T H R E A D ---------------
> > Current thread (0x0000000047276800): GCTaskThread [stack:
> > 0x0000000000000000,0x0000000000000000] [id=974]
> > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
> > si_addr=0x0000000000000000
> >
> > --------------- P R O C E S S ---------------
> > Java Threads: ( => current thread )
> > 0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1"
> [_thread_blocked,
> > id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
> > 0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
> > [_thread_blocked, id=1270, stack(0x0000000049dbf000,0x0000000049ec0000)]
> > 0x0000000047691000 JavaThread "Thread-26" [_thread_in_native, id=1191,
> > stack(0x0000000043d53000,0x0000000043e54000)]
> > 0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native, id=1117,
> > stack(0x0000000048daf000,0x0000000048eb0000)]
> > 0x0000000047a42800 JavaThread "WRITE-/192.168.252.61"
> [_thread_blocked,
> > id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
> > 0x0000000047abc800 JavaThread "WRITE-/192.168.252.62"
> [_thread_blocked,
> > id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
> > 0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked,
> id=1108,
> > stack(0x0000000049abc000,0x0000000049bbd000)]
> > 0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked,
> > id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
> > 0x0000000047693800 JavaThread "Thread-16" [_thread_in_native, id=1106,
> > stack(0x00000000498ba000,0x00000000499bb000)]
> > 0x0000000047693000 JavaThread "WRITE-/192.168.252.99"
> [_thread_blocked,
> > id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
> > 0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
> > stack(0x00000000496b8000,0x00000000497b9000)]
> > 0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native, id=1103,
> > stack(0x00000000495b7000,0x00000000496b8000)]
> > 0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59"
> [_thread_blocked,
> > id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
> > 0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native, id=1101,
> > stack(0x00000000493b5000,0x00000000494b6000)]
> > 0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61"
> [_thread_blocked,
> > id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
> > 0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58"
> [_thread_blocked,
> > id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
> > 0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62"
> [_thread_blocked,
> > id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
> > 0x00000000475db000 JavaThread "Thread-12" [_thread_in_native, id=1097,
> > stack(0x0000000048fb1000,0x00000000490b2000)]
> > 0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
> > stack(0x0000000048eb0000,0x0000000048fb1000)]
> > 0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
> > stack(0x0000000048bad000,0x0000000048cae000)]
> > 0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
> > stack(0x0000000048aac000,0x0000000048bad000)]
> > 0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked, id=1091,
> > stack(0x00000000489ab000,0x0000000048aac000)]
> > 0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked,
> id=1090,
> > stack(0x00000000488aa000,0x00000000489ab000)]
> > 0x00002aabbc1ac000 JavaThread
> > "ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
> > id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
> > 0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked,
> id=1088,
> > stack(0x0000000040694000,0x0000000040795000)]
> > 0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
> > id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
> > 0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
> > [_thread_blocked, id=1086, stack(0x00000000485a7000,0x00000000486a8000)]
> > 0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
> > id=1085, stack(0x0000000040830000,0x0000000040931000)]
> > 0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked,
> > id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
> > 0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
> > id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
> > 0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
> > stack(0x00000000482a4000,0x00000000483a5000)]
> > 0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked,
> id=1078,
> > stack(0x00000000481a3000,0x00000000482a4000)]
> > 0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
> > id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
> > 0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
> > id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
> > 0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
> > id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
> > 0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
> > id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
> > 0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
> > id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
> > 0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
> > id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
> > 0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
> > id=1071, stack(0x0000000047086000,0x0000000047187000)]
> > 0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
> > id=1070, stack(0x0000000046f85000,0x0000000047086000)]
> > 0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
> > id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
> > 0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
> > id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
> > 0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
> > id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
> > 0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
> > id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
> > 0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
> > id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
> > 0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
> > id=1064, stack(0x000000004697f000,0x0000000046a80000)]
> > 0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
> > id=1063, stack(0x000000004687e000,0x000000004697f000)]
> > 0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
> > id=1062, stack(0x000000004677d000,0x000000004687e000)]
> > 0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
> > id=1061, stack(0x000000004667c000,0x000000004677d000)]
> > 0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
> > id=1060, stack(0x000000004657b000,0x000000004667c000)]
> > 0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
> > id=1059, stack(0x000000004647a000,0x000000004657b000)]
> > 0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
> > id=1058, stack(0x0000000046379000,0x000000004647a000)]
> > 0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
> > id=1057, stack(0x0000000046278000,0x0000000046379000)]
> > 0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
> > id=1056, stack(0x0000000046177000,0x0000000046278000)]
> > 0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
> > id=1055, stack(0x0000000046076000,0x0000000046177000)]
> > 0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
> > id=1054, stack(0x0000000045f75000,0x0000000046076000)]
> > 0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked,
> > id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
> > 0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked,
> > id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
> > 0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked,
> > id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
> > 0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked,
> > id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
> > 0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked,
> > id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
> > 0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked,
> > id=1048, stack(0x000000004596f000,0x0000000045a70000)]
> > 0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked,
> > id=1047, stack(0x000000004586e000,0x000000004596f000)]
> > 0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked,
> > id=1046, stack(0x000000004576d000,0x000000004586e000)]
> > 0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked,
> > id=1045, stack(0x000000004566c000,0x000000004576d000)]
> > 0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked,
> > id=1044, stack(0x000000004556b000,0x000000004566c000)]
> > 0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked,
> > id=1043, stack(0x000000004546a000,0x000000004556b000)]
> > 0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked,
> > id=1042, stack(0x0000000045369000,0x000000004546a000)]
> > 0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked,
> > id=1041, stack(0x0000000045268000,0x0000000045369000)]
> > 0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked,
> > id=1040, stack(0x0000000045167000,0x0000000045268000)]
> > 0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked,
> > id=1039, stack(0x0000000045066000,0x0000000045167000)]
> > 0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked,
> > id=1038, stack(0x0000000044f65000,0x0000000045066000)]
> > 0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked,
> > id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
> > 0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked,
> > id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
> > 0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked,
> > id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
> > 0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked,
> > id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
> > 0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked,
> > id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
> > 0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked,
> > id=1032, stack(0x000000004495f000,0x0000000044a60000)]
> > 0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked,
> > id=1031, stack(0x000000004485e000,0x000000004495f000)]
> > 0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
> > id=1030, stack(0x000000004475d000,0x000000004485e000)]
> > 0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
> > id=1029, stack(0x000000004465c000,0x000000004475d000)]
> > 0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
> > id=1028, stack(0x000000004455b000,0x000000004465c000)]
> > 0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
> > id=1027, stack(0x000000004445a000,0x000000004455b000)]
> > 0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
> > id=1026, stack(0x0000000044359000,0x000000004445a000)]
> > 0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
> > id=1025, stack(0x0000000044258000,0x0000000044359000)]
> > 0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
> > id=1024, stack(0x0000000044157000,0x0000000044258000)]
> > 0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
> > id=1023, stack(0x0000000044056000,0x0000000044157000)]
> > 0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
> > id=1022, stack(0x0000000043f55000,0x0000000044056000)]
> > 0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked,
> > id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
> > 0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon
> [_thread_blocked,
> > id=1017, stack(0x0000000041499000,0x000000004159a000)]
> > 0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked,
> id=1009,
> > stack(0x0000000043c52000,0x0000000043d53000)]
> > 0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1"
> [_thread_blocked,
> > id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
> > 0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
> > [_thread_blocked, id=1007, stack(0x0000000043a50000,0x0000000043b51000)]
> > 0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
> > [_thread_blocked, id=1006, stack(0x000000004394f000,0x0000000043a50000)]
> > 0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
> > [_thread_blocked, id=1005, stack(0x000000004384e000,0x000000004394f000)]
> > 0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked,
> > id=1004, stack(0x000000004374d000,0x000000004384e000)]
> > 0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked,
> > id=1003, stack(0x000000004364c000,0x000000004374d000)]
> > 0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked,
> > id=1002, stack(0x000000004354b000,0x000000004364c000)]
> > 0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked,
> > id=1001, stack(0x000000004344a000,0x000000004354b000)]
> > 0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1"
> [_thread_blocked,
> > id=1000, stack(0x0000000043349000,0x000000004344a000)]
> > 0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
> > id=999, stack(0x0000000043248000,0x0000000043349000)]
> > 0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
> > id=998, stack(0x0000000043147000,0x0000000043248000)]
> > 0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked, id=997,
> > stack(0x000000004117f000,0x0000000041280000)]
> > 0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
> > [_thread_in_native, id=995, stack(0x0000000040f7d000,0x000000004107e000)]
> > 0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
> > [_thread_in_native, id=994, stack(0x0000000043046000,0x0000000043147000)]
> > 0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
> > [_thread_in_native, id=989, stack(0x0000000040a48000,0x0000000040b49000)]
> > 0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
> > [_thread_blocked, id=988, stack(0x0000000042f45000,0x0000000043046000)]
> > 0x00002aabbc024000 JavaThread "CompilerThread1" daemon
> [_thread_blocked,
> > id=987, stack(0x0000000042e44000,0x0000000042f45000)]
> > 0x00002aabbc022000 JavaThread "CompilerThread0" daemon
> [_thread_blocked,
> > id=986, stack(0x0000000042d43000,0x0000000042e44000)]
> > 0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon
> [_thread_blocked,
> > id=985, stack(0x00000000417e8000,0x00000000418e9000)]
> > 0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
> > [_thread_blocked, id=984, stack(0x00000000416e7000,0x00000000417e8000)]
> > 0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked,
> id=983,
> > stack(0x0000000042c42000,0x0000000042d43000)]
> > 0x0000000047424800 JavaThread "Reference Handler" daemon
> [_thread_blocked,
> > id=982, stack(0x0000000042b41000,0x0000000042c42000)]
> > 0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
> > stack(0x00000000412b1000,0x00000000413b2000)]
> > Other Threads:
> > 0x000000004741d800 VMThread [stack:
> 0x0000000042a40000,0x0000000042b41000]
> > [id=981]
> > 0x00002aabbc181000 WatcherThread [stack:
> > 0x000000004107e000,0x000000004117f000] [id=996]
> > =>0x0000000047276800 (exited) GCTaskThread [stack:
> > 0x0000000000000000,0x0000000000000000] [id=974]
> > VM state:at safepoint (normal execution)
> > VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> > [0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
> > [0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
> > Heap
> > par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
> > 0x00002aaabf050000, 0x00002aaabf050000)
> > eden space 272640K, 100% used [0x00002aaaae1f0000, 0x00002aaabec30000,
> > 0x00002aaabec30000)
> > from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
> > 0x00002aaabf050000)
> > to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
> > 0x00002aaabee40000)
> > concurrent mark-sweep generation total 3184972K, used 2657590K
> > [0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
> > concurrent-mark-sweep perm gen total 26272K, used 15697K
> > [0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
> > VM Arguments:
> > jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90
> -XX:+AggressiveOpts
> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> > -XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
> > -XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
> > -Dcassandra-pidfile=/var/run/cassandra.pid
> > java_command: org.apache.cassandra.thrift.CassandraDaemon
> > Launcher Type: SUN_STANDARD
> > Environment Variables:
> > JAVA_HOME=/usr/java/jdk1.6.0_17/
> >
> PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
> >
> LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
> > SHELL=/bin/bash
> > Signal Handlers:
> > SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000, sa_flags=0x10000004
> > SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> >
> > --------------- S Y S T E M ---------------
> > OS:CentOS release 5.4 (Final)
> > uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010
> x86_64
> > libc:glibc 2.5 NPTL 2.5
> > rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS infinity
> > load average:2.50 3.93 5.66
> > CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
> > stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2,
> ht
> > Memory: 4k page, physical 16429856k(55128k free), swap 5242872k(5220216k
> > free)
> > vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64 JRE
> > (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc 3.2.2
> > (SuSE Linux)
> > time: Tue May 4 17:03:44 2010
> > elapsed time: 11909 seconds
> >
> >
>
Re: cassandra jvm crash in GCTaskThread
Posted by Ran Tavory <ra...@gmail.com>.
ok thanks, I'm trying _20
On Wed, May 5, 2010 at 5:24 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Yes, those are the recommended versions (_20 is just _19 with some
> security fix that doesn't affect Cassandra).
>
> On Wed, May 5, 2010 at 2:33 AM, Ran Tavory <ra...@gmail.com> wrote:
> > has anyone tried jvm _20 or _19 with cassandra?
> >
> > On Wed, May 5, 2010 at 10:00 AM, Nathan McCall <na...@vervewireless.com>
> > wrote:
> >>
> >> Ran,
> >> You may want to upgrade your jvm. I had a similar core with a heavily
> >> loaded tomcat recently and found the following on the release notes
> >> page for 1.6.0_18:
> >> "Card-Marking Optimization Issue
> >>
> >> A flaw in the implementation of a card-marking performance
> >> optimization in the JVM can cause heap corruption under some
> >> circumstances. This issue affects the CMS garbage collector prior to
> >> 6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The
> >> serial garbage collector is not affected. Applications most likely to
> >> be affected by this issue are those that allocate very large objects
> >> which would not normally fit in Eden, or those that make extensive use
> >> of JNI Critical Sections (JNI Get/Release*Critical)."
> >>
> >> http://java.sun.com/javase/6/webnotes/6u18.html
> >>
> >> Note that there is an apparent work-around jvm option to add:
> >> -XX:-ReduceInitialCardMarks
> >>
> >> -Nate
> >>
> >>
> >> On Tue, May 4, 2010 at 11:24 PM, Ran Tavory <ra...@gmail.com> wrote:
> >> > Running a cluster of 0.6.1, one of the hosts crashed during GC.
> >> > Can anyone suggest what needs tweaking to prevent that?
> >> > The last configuration change I made was to
> >> > change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I
> >> > can't
> >> > determine if it's related.
> >> >
> >> > System is 64 bit centos, the rest of the system info is at the bottom
> of
> >> > the
> >> > email.
> >> > The last lines before the crash from system.log are:
> >> >
> >> > INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913
> CompactionManager.java
> >> > (line 246) Compacting
> >> >
> >> >
> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
> >> > INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line
> >> > 407)
> >> > Discarding obsolete commit
> >> >
> >> >
> log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
> >> > INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820
> CompactionManager.java
> >> > (line 326) Compacted to
> >> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
> >> > 291072443/291071793 bytes for 123292 keys. Time: 19906ms.
> >> > And a summary of hs_err is here. The rest is attached. Let me know if
> >> > you've
> >> > see this before and can advice, thanks!
> >> > #
> >> > # A fatal error has been detected by the Java Runtime Environment:
> >> > #
> >> > # SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
> >> > #
> >> > # JRE version: 6.0_17-b04
> >> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
> >> > linux-amd64 )
> >> > # Problematic frame:
> >> > # V [libjvm.so+0x1ffb76]
> >> > #
> >> > # If you would like to submit a bug report, please visit:
> >> > # http://java.sun.com/webapps/bugreport/crash.jsp
> >> > #
> >> > --------------- T H R E A D ---------------
> >> > Current thread (0x0000000047276800): GCTaskThread [stack:
> >> > 0x0000000000000000,0x0000000000000000] [id=974]
> >> > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
> >> > si_addr=0x0000000000000000
> >> >
> >> > --------------- P R O C E S S ---------------
> >> > Java Threads: ( => current thread )
> >> > 0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1"
> >> > [_thread_blocked,
> >> > id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
> >> > 0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
> >> > [_thread_blocked, id=1270,
> stack(0x0000000049dbf000,0x0000000049ec0000)]
> >> > 0x0000000047691000 JavaThread "Thread-26" [_thread_in_native,
> id=1191,
> >> > stack(0x0000000043d53000,0x0000000043e54000)]
> >> > 0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native,
> id=1117,
> >> > stack(0x0000000048daf000,0x0000000048eb0000)]
> >> > 0x0000000047a42800 JavaThread "WRITE-/192.168.252.61"
> >> > [_thread_blocked,
> >> > id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
> >> > 0x0000000047abc800 JavaThread "WRITE-/192.168.252.62"
> >> > [_thread_blocked,
> >> > id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
> >> > 0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked,
> >> > id=1108,
> >> > stack(0x0000000049abc000,0x0000000049bbd000)]
> >> > 0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1"
> >> > [_thread_blocked,
> >> > id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
> >> > 0x0000000047693800 JavaThread "Thread-16" [_thread_in_native,
> id=1106,
> >> > stack(0x00000000498ba000,0x00000000499bb000)]
> >> > 0x0000000047693000 JavaThread "WRITE-/192.168.252.99"
> >> > [_thread_blocked,
> >> > id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
> >> > 0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
> >> > stack(0x00000000496b8000,0x00000000497b9000)]
> >> > 0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native,
> id=1103,
> >> > stack(0x00000000495b7000,0x00000000496b8000)]
> >> > 0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59"
> >> > [_thread_blocked,
> >> > id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
> >> > 0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native,
> id=1101,
> >> > stack(0x00000000493b5000,0x00000000494b6000)]
> >> > 0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61"
> >> > [_thread_blocked,
> >> > id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
> >> > 0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58"
> >> > [_thread_blocked,
> >> > id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
> >> > 0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62"
> >> > [_thread_blocked,
> >> > id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
> >> > 0x00000000475db000 JavaThread "Thread-12" [_thread_in_native,
> id=1097,
> >> > stack(0x0000000048fb1000,0x00000000490b2000)]
> >> > 0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
> >> > stack(0x0000000048eb0000,0x0000000048fb1000)]
> >> > 0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
> >> > stack(0x0000000048bad000,0x0000000048cae000)]
> >> > 0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
> >> > stack(0x0000000048aac000,0x0000000048bad000)]
> >> > 0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked,
> id=1091,
> >> > stack(0x00000000489ab000,0x0000000048aac000)]
> >> > 0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked,
> >> > id=1090,
> >> > stack(0x00000000488aa000,0x00000000489ab000)]
> >> > 0x00002aabbc1ac000 JavaThread
> >> > "ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
> >> > id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
> >> > 0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked,
> >> > id=1088,
> >> > stack(0x0000000040694000,0x0000000040795000)]
> >> > 0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
> >> > id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
> >> > 0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
> >> > [_thread_blocked, id=1086,
> stack(0x00000000485a7000,0x00000000486a8000)]
> >> > 0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
> >> > id=1085, stack(0x0000000040830000,0x0000000040931000)]
> >> > 0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1"
> >> > [_thread_blocked,
> >> > id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
> >> > 0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
> >> > id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
> >> > 0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
> >> > stack(0x00000000482a4000,0x00000000483a5000)]
> >> > 0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked,
> >> > id=1078,
> >> > stack(0x00000000481a3000,0x00000000482a4000)]
> >> > 0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
> >> > id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
> >> > 0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
> >> > id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
> >> > 0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
> >> > id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
> >> > 0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
> >> > id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
> >> > 0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
> >> > id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
> >> > 0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
> >> > id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
> >> > 0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
> >> > id=1071, stack(0x0000000047086000,0x0000000047187000)]
> >> > 0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
> >> > id=1070, stack(0x0000000046f85000,0x0000000047086000)]
> >> > 0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
> >> > id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
> >> > 0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
> >> > id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
> >> > 0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
> >> > id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
> >> > 0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
> >> > id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
> >> > 0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
> >> > id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
> >> > 0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
> >> > id=1064, stack(0x000000004697f000,0x0000000046a80000)]
> >> > 0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
> >> > id=1063, stack(0x000000004687e000,0x000000004697f000)]
> >> > 0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
> >> > id=1062, stack(0x000000004677d000,0x000000004687e000)]
> >> > 0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
> >> > id=1061, stack(0x000000004667c000,0x000000004677d000)]
> >> > 0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
> >> > id=1060, stack(0x000000004657b000,0x000000004667c000)]
> >> > 0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
> >> > id=1059, stack(0x000000004647a000,0x000000004657b000)]
> >> > 0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
> >> > id=1058, stack(0x0000000046379000,0x000000004647a000)]
> >> > 0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
> >> > id=1057, stack(0x0000000046278000,0x0000000046379000)]
> >> > 0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
> >> > id=1056, stack(0x0000000046177000,0x0000000046278000)]
> >> > 0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
> >> > id=1055, stack(0x0000000046076000,0x0000000046177000)]
> >> > 0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
> >> > id=1054, stack(0x0000000045f75000,0x0000000046076000)]
> >> > 0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32"
> >> > [_thread_blocked,
> >> > id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
> >> > 0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31"
> >> > [_thread_blocked,
> >> > id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
> >> > 0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30"
> >> > [_thread_blocked,
> >> > id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
> >> > 0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29"
> >> > [_thread_blocked,
> >> > id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
> >> > 0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28"
> >> > [_thread_blocked,
> >> > id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
> >> > 0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27"
> >> > [_thread_blocked,
> >> > id=1048, stack(0x000000004596f000,0x0000000045a70000)]
> >> > 0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26"
> >> > [_thread_blocked,
> >> > id=1047, stack(0x000000004586e000,0x000000004596f000)]
> >> > 0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25"
> >> > [_thread_blocked,
> >> > id=1046, stack(0x000000004576d000,0x000000004586e000)]
> >> > 0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24"
> >> > [_thread_blocked,
> >> > id=1045, stack(0x000000004566c000,0x000000004576d000)]
> >> > 0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23"
> >> > [_thread_blocked,
> >> > id=1044, stack(0x000000004556b000,0x000000004566c000)]
> >> > 0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22"
> >> > [_thread_blocked,
> >> > id=1043, stack(0x000000004546a000,0x000000004556b000)]
> >> > 0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21"
> >> > [_thread_blocked,
> >> > id=1042, stack(0x0000000045369000,0x000000004546a000)]
> >> > 0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20"
> >> > [_thread_blocked,
> >> > id=1041, stack(0x0000000045268000,0x0000000045369000)]
> >> > 0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19"
> >> > [_thread_blocked,
> >> > id=1040, stack(0x0000000045167000,0x0000000045268000)]
> >> > 0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18"
> >> > [_thread_blocked,
> >> > id=1039, stack(0x0000000045066000,0x0000000045167000)]
> >> > 0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17"
> >> > [_thread_blocked,
> >> > id=1038, stack(0x0000000044f65000,0x0000000045066000)]
> >> > 0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16"
> >> > [_thread_blocked,
> >> > id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
> >> > 0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15"
> >> > [_thread_blocked,
> >> > id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
> >> > 0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14"
> >> > [_thread_blocked,
> >> > id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
> >> > 0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13"
> >> > [_thread_blocked,
> >> > id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
> >> > 0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12"
> >> > [_thread_blocked,
> >> > id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
> >> > 0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11"
> >> > [_thread_blocked,
> >> > id=1032, stack(0x000000004495f000,0x0000000044a60000)]
> >> > 0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10"
> >> > [_thread_blocked,
> >> > id=1031, stack(0x000000004485e000,0x000000004495f000)]
> >> > 0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9"
> [_thread_blocked,
> >> > id=1030, stack(0x000000004475d000,0x000000004485e000)]
> >> > 0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8"
> [_thread_blocked,
> >> > id=1029, stack(0x000000004465c000,0x000000004475d000)]
> >> > 0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7"
> [_thread_blocked,
> >> > id=1028, stack(0x000000004455b000,0x000000004465c000)]
> >> > 0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6"
> [_thread_blocked,
> >> > id=1027, stack(0x000000004445a000,0x000000004455b000)]
> >> > 0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5"
> [_thread_blocked,
> >> > id=1026, stack(0x0000000044359000,0x000000004445a000)]
> >> > 0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4"
> [_thread_blocked,
> >> > id=1025, stack(0x0000000044258000,0x0000000044359000)]
> >> > 0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3"
> [_thread_blocked,
> >> > id=1024, stack(0x0000000044157000,0x0000000044258000)]
> >> > 0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2"
> [_thread_blocked,
> >> > id=1023, stack(0x0000000044056000,0x0000000044157000)]
> >> > 0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1"
> [_thread_blocked,
> >> > id=1022, stack(0x0000000043f55000,0x0000000044056000)]
> >> > 0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER"
> >> > [_thread_blocked,
> >> > id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
> >> > 0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon
> >> > [_thread_blocked,
> >> > id=1017, stack(0x0000000041499000,0x000000004159a000)]
> >> > 0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked,
> >> > id=1009,
> >> > stack(0x0000000043c52000,0x0000000043d53000)]
> >> > 0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1"
> >> > [_thread_blocked,
> >> > id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
> >> > 0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
> >> > [_thread_blocked, id=1007,
> stack(0x0000000043a50000,0x0000000043b51000)]
> >> > 0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
> >> > [_thread_blocked, id=1006,
> stack(0x000000004394f000,0x0000000043a50000)]
> >> > 0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
> >> > [_thread_blocked, id=1005,
> stack(0x000000004384e000,0x000000004394f000)]
> >> > 0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4"
> >> > [_thread_blocked,
> >> > id=1004, stack(0x000000004374d000,0x000000004384e000)]
> >> > 0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3"
> >> > [_thread_blocked,
> >> > id=1003, stack(0x000000004364c000,0x000000004374d000)]
> >> > 0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2"
> >> > [_thread_blocked,
> >> > id=1002, stack(0x000000004354b000,0x000000004364c000)]
> >> > 0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1"
> >> > [_thread_blocked,
> >> > id=1001, stack(0x000000004344a000,0x000000004354b000)]
> >> > 0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1"
> >> > [_thread_blocked,
> >> > id=1000, stack(0x0000000043349000,0x000000004344a000)]
> >> > 0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1"
> [_thread_blocked,
> >> > id=999, stack(0x0000000043248000,0x0000000043349000)]
> >> > 0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1"
> [_thread_blocked,
> >> > id=998, stack(0x0000000043147000,0x0000000043248000)]
> >> > 0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked,
> id=997,
> >> > stack(0x000000004117f000,0x0000000041280000)]
> >> > 0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
> >> > [_thread_in_native, id=995,
> >> > stack(0x0000000040f7d000,0x000000004107e000)]
> >> > 0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
> >> > [_thread_in_native, id=994,
> >> > stack(0x0000000043046000,0x0000000043147000)]
> >> > 0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
> >> > [_thread_in_native, id=989,
> >> > stack(0x0000000040a48000,0x0000000040b49000)]
> >> > 0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
> >> > [_thread_blocked, id=988,
> stack(0x0000000042f45000,0x0000000043046000)]
> >> > 0x00002aabbc024000 JavaThread "CompilerThread1" daemon
> >> > [_thread_blocked,
> >> > id=987, stack(0x0000000042e44000,0x0000000042f45000)]
> >> > 0x00002aabbc022000 JavaThread "CompilerThread0" daemon
> >> > [_thread_blocked,
> >> > id=986, stack(0x0000000042d43000,0x0000000042e44000)]
> >> > 0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon
> >> > [_thread_blocked,
> >> > id=985, stack(0x00000000417e8000,0x00000000418e9000)]
> >> > 0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
> >> > [_thread_blocked, id=984,
> stack(0x00000000416e7000,0x00000000417e8000)]
> >> > 0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked,
> >> > id=983,
> >> > stack(0x0000000042c42000,0x0000000042d43000)]
> >> > 0x0000000047424800 JavaThread "Reference Handler" daemon
> >> > [_thread_blocked,
> >> > id=982, stack(0x0000000042b41000,0x0000000042c42000)]
> >> > 0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
> >> > stack(0x00000000412b1000,0x00000000413b2000)]
> >> > Other Threads:
> >> > 0x000000004741d800 VMThread [stack:
> >> > 0x0000000042a40000,0x0000000042b41000]
> >> > [id=981]
> >> > 0x00002aabbc181000 WatcherThread [stack:
> >> > 0x000000004107e000,0x000000004117f000] [id=996]
> >> > =>0x0000000047276800 (exited) GCTaskThread [stack:
> >> > 0x0000000000000000,0x0000000000000000] [id=974]
> >> > VM state:at safepoint (normal execution)
> >> > VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> >> > [0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
> >> > [0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
> >> > Heap
> >> > par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
> >> > 0x00002aaabf050000, 0x00002aaabf050000)
> >> > eden space 272640K, 100% used [0x00002aaaae1f0000,
> 0x00002aaabec30000,
> >> > 0x00002aaabec30000)
> >> > from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
> >> > 0x00002aaabf050000)
> >> > to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
> >> > 0x00002aaabee40000)
> >> > concurrent mark-sweep generation total 3184972K, used 2657590K
> >> > [0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
> >> > concurrent-mark-sweep perm gen total 26272K, used 15697K
> >> > [0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
> >> > VM Arguments:
> >> > jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90
> >> > -XX:+AggressiveOpts
> >> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> >> > -XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
> >> > -XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
> >> > -Dcom.sun.management.jmxremote.ssl=false
> >> > -Dcom.sun.management.jmxremote.authenticate=false
> >> >
> -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
> >> > -Dcassandra-pidfile=/var/run/cassandra.pid
> >> > java_command: org.apache.cassandra.thrift.CassandraDaemon
> >> > Launcher Type: SUN_STANDARD
> >> > Environment Variables:
> >> > JAVA_HOME=/usr/java/jdk1.6.0_17/
> >> >
> >> >
> PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
> >> >
> >> >
> LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
> >> > SHELL=/bin/bash
> >> > Signal Handlers:
> >> > SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff,
> >> > sa_flags=0x10000004
> >> > SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff,
> sa_flags=0x10000004
> >> > SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
> sa_flags=0x10000004
> >> > SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
> >> > sa_flags=0x10000004
> >> > SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
> >> > sa_flags=0x10000004
> >> > SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
> sa_flags=0x10000004
> >> > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> >> > SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000,
> >> > sa_flags=0x10000004
> >> > SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff,
> sa_flags=0x10000004
> >> > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> >> > SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff,
> >> > sa_flags=0x10000004
> >> > SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff,
> >> > sa_flags=0x10000004
> >> >
> >> > --------------- S Y S T E M ---------------
> >> > OS:CentOS release 5.4 (Final)
> >> > uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010
> >> > x86_64
> >> > libc:glibc 2.5 NPTL 2.5
> >> > rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS
> infinity
> >> > load average:2.50 3.93 5.66
> >> > CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
> >> > stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1,
> >> > sse4.2, ht
> >> > Memory: 4k page, physical 16429856k(55128k free), swap
> 5242872k(5220216k
> >> > free)
> >> > vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64
> >> > JRE
> >> > (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc
> >> > 3.2.2
> >> > (SuSE Linux)
> >> > time: Tue May 4 17:03:44 2010
> >> > elapsed time: 11909 seconds
> >> >
> >> >
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: cassandra jvm crash in GCTaskThread
Posted by Jonathan Ellis <jb...@gmail.com>.
Yes, those are the recommended versions (_20 is just _19 with some
security fix that doesn't affect Cassandra).
On Wed, May 5, 2010 at 2:33 AM, Ran Tavory <ra...@gmail.com> wrote:
> has anyone tried jvm _20 or _19 with cassandra?
>
> On Wed, May 5, 2010 at 10:00 AM, Nathan McCall <na...@vervewireless.com>
> wrote:
>>
>> Ran,
>> You may want to upgrade your jvm. I had a similar core with a heavily
>> loaded tomcat recently and found the following on the release notes
>> page for 1.6.0_18:
>> "Card-Marking Optimization Issue
>>
>> A flaw in the implementation of a card-marking performance
>> optimization in the JVM can cause heap corruption under some
>> circumstances. This issue affects the CMS garbage collector prior to
>> 6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The
>> serial garbage collector is not affected. Applications most likely to
>> be affected by this issue are those that allocate very large objects
>> which would not normally fit in Eden, or those that make extensive use
>> of JNI Critical Sections (JNI Get/Release*Critical)."
>>
>> http://java.sun.com/javase/6/webnotes/6u18.html
>>
>> Note that there is an apparent work-around jvm option to add:
>> -XX:-ReduceInitialCardMarks
>>
>> -Nate
>>
>>
>> On Tue, May 4, 2010 at 11:24 PM, Ran Tavory <ra...@gmail.com> wrote:
>> > Running a cluster of 0.6.1, one of the hosts crashed during GC.
>> > Can anyone suggest what needs tweaking to prevent that?
>> > The last configuration change I made was to
>> > change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I
>> > can't
>> > determine if it's related.
>> >
>> > System is 64 bit centos, the rest of the system info is at the bottom of
>> > the
>> > email.
>> > The last lines before the crash from system.log are:
>> >
>> > INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913 CompactionManager.java
>> > (line 246) Compacting
>> >
>> > [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
>> > INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line
>> > 407)
>> > Discarding obsolete commit
>> >
>> > log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
>> > INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820 CompactionManager.java
>> > (line 326) Compacted to
>> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
>> > 291072443/291071793 bytes for 123292 keys. Time: 19906ms.
>> > And a summary of hs_err is here. The rest is attached. Let me know if
>> > you've
>> > see this before and can advice, thanks!
>> > #
>> > # A fatal error has been detected by the Java Runtime Environment:
>> > #
>> > # SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
>> > #
>> > # JRE version: 6.0_17-b04
>> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
>> > linux-amd64 )
>> > # Problematic frame:
>> > # V [libjvm.so+0x1ffb76]
>> > #
>> > # If you would like to submit a bug report, please visit:
>> > # http://java.sun.com/webapps/bugreport/crash.jsp
>> > #
>> > --------------- T H R E A D ---------------
>> > Current thread (0x0000000047276800): GCTaskThread [stack:
>> > 0x0000000000000000,0x0000000000000000] [id=974]
>> > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
>> > si_addr=0x0000000000000000
>> >
>> > --------------- P R O C E S S ---------------
>> > Java Threads: ( => current thread )
>> > 0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1"
>> > [_thread_blocked,
>> > id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
>> > 0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
>> > [_thread_blocked, id=1270, stack(0x0000000049dbf000,0x0000000049ec0000)]
>> > 0x0000000047691000 JavaThread "Thread-26" [_thread_in_native, id=1191,
>> > stack(0x0000000043d53000,0x0000000043e54000)]
>> > 0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native, id=1117,
>> > stack(0x0000000048daf000,0x0000000048eb0000)]
>> > 0x0000000047a42800 JavaThread "WRITE-/192.168.252.61"
>> > [_thread_blocked,
>> > id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
>> > 0x0000000047abc800 JavaThread "WRITE-/192.168.252.62"
>> > [_thread_blocked,
>> > id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
>> > 0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked,
>> > id=1108,
>> > stack(0x0000000049abc000,0x0000000049bbd000)]
>> > 0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1"
>> > [_thread_blocked,
>> > id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
>> > 0x0000000047693800 JavaThread "Thread-16" [_thread_in_native, id=1106,
>> > stack(0x00000000498ba000,0x00000000499bb000)]
>> > 0x0000000047693000 JavaThread "WRITE-/192.168.252.99"
>> > [_thread_blocked,
>> > id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
>> > 0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
>> > stack(0x00000000496b8000,0x00000000497b9000)]
>> > 0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native, id=1103,
>> > stack(0x00000000495b7000,0x00000000496b8000)]
>> > 0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59"
>> > [_thread_blocked,
>> > id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
>> > 0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native, id=1101,
>> > stack(0x00000000493b5000,0x00000000494b6000)]
>> > 0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61"
>> > [_thread_blocked,
>> > id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
>> > 0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58"
>> > [_thread_blocked,
>> > id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
>> > 0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62"
>> > [_thread_blocked,
>> > id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
>> > 0x00000000475db000 JavaThread "Thread-12" [_thread_in_native, id=1097,
>> > stack(0x0000000048fb1000,0x00000000490b2000)]
>> > 0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
>> > stack(0x0000000048eb0000,0x0000000048fb1000)]
>> > 0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
>> > stack(0x0000000048bad000,0x0000000048cae000)]
>> > 0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
>> > stack(0x0000000048aac000,0x0000000048bad000)]
>> > 0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked, id=1091,
>> > stack(0x00000000489ab000,0x0000000048aac000)]
>> > 0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked,
>> > id=1090,
>> > stack(0x00000000488aa000,0x00000000489ab000)]
>> > 0x00002aabbc1ac000 JavaThread
>> > "ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
>> > id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
>> > 0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked,
>> > id=1088,
>> > stack(0x0000000040694000,0x0000000040795000)]
>> > 0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
>> > id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
>> > 0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
>> > [_thread_blocked, id=1086, stack(0x00000000485a7000,0x00000000486a8000)]
>> > 0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
>> > id=1085, stack(0x0000000040830000,0x0000000040931000)]
>> > 0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1"
>> > [_thread_blocked,
>> > id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
>> > 0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
>> > id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
>> > 0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
>> > stack(0x00000000482a4000,0x00000000483a5000)]
>> > 0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked,
>> > id=1078,
>> > stack(0x00000000481a3000,0x00000000482a4000)]
>> > 0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
>> > id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
>> > 0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
>> > id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
>> > 0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
>> > id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
>> > 0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
>> > id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
>> > 0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
>> > id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
>> > 0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
>> > id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
>> > 0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
>> > id=1071, stack(0x0000000047086000,0x0000000047187000)]
>> > 0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
>> > id=1070, stack(0x0000000046f85000,0x0000000047086000)]
>> > 0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
>> > id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
>> > 0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
>> > id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
>> > 0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
>> > id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
>> > 0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
>> > id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
>> > 0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
>> > id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
>> > 0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
>> > id=1064, stack(0x000000004697f000,0x0000000046a80000)]
>> > 0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
>> > id=1063, stack(0x000000004687e000,0x000000004697f000)]
>> > 0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
>> > id=1062, stack(0x000000004677d000,0x000000004687e000)]
>> > 0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
>> > id=1061, stack(0x000000004667c000,0x000000004677d000)]
>> > 0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
>> > id=1060, stack(0x000000004657b000,0x000000004667c000)]
>> > 0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
>> > id=1059, stack(0x000000004647a000,0x000000004657b000)]
>> > 0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
>> > id=1058, stack(0x0000000046379000,0x000000004647a000)]
>> > 0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
>> > id=1057, stack(0x0000000046278000,0x0000000046379000)]
>> > 0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
>> > id=1056, stack(0x0000000046177000,0x0000000046278000)]
>> > 0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
>> > id=1055, stack(0x0000000046076000,0x0000000046177000)]
>> > 0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
>> > id=1054, stack(0x0000000045f75000,0x0000000046076000)]
>> > 0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32"
>> > [_thread_blocked,
>> > id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
>> > 0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31"
>> > [_thread_blocked,
>> > id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
>> > 0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30"
>> > [_thread_blocked,
>> > id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
>> > 0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29"
>> > [_thread_blocked,
>> > id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
>> > 0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28"
>> > [_thread_blocked,
>> > id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
>> > 0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27"
>> > [_thread_blocked,
>> > id=1048, stack(0x000000004596f000,0x0000000045a70000)]
>> > 0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26"
>> > [_thread_blocked,
>> > id=1047, stack(0x000000004586e000,0x000000004596f000)]
>> > 0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25"
>> > [_thread_blocked,
>> > id=1046, stack(0x000000004576d000,0x000000004586e000)]
>> > 0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24"
>> > [_thread_blocked,
>> > id=1045, stack(0x000000004566c000,0x000000004576d000)]
>> > 0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23"
>> > [_thread_blocked,
>> > id=1044, stack(0x000000004556b000,0x000000004566c000)]
>> > 0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22"
>> > [_thread_blocked,
>> > id=1043, stack(0x000000004546a000,0x000000004556b000)]
>> > 0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21"
>> > [_thread_blocked,
>> > id=1042, stack(0x0000000045369000,0x000000004546a000)]
>> > 0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20"
>> > [_thread_blocked,
>> > id=1041, stack(0x0000000045268000,0x0000000045369000)]
>> > 0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19"
>> > [_thread_blocked,
>> > id=1040, stack(0x0000000045167000,0x0000000045268000)]
>> > 0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18"
>> > [_thread_blocked,
>> > id=1039, stack(0x0000000045066000,0x0000000045167000)]
>> > 0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17"
>> > [_thread_blocked,
>> > id=1038, stack(0x0000000044f65000,0x0000000045066000)]
>> > 0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16"
>> > [_thread_blocked,
>> > id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
>> > 0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15"
>> > [_thread_blocked,
>> > id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
>> > 0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14"
>> > [_thread_blocked,
>> > id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
>> > 0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13"
>> > [_thread_blocked,
>> > id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
>> > 0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12"
>> > [_thread_blocked,
>> > id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
>> > 0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11"
>> > [_thread_blocked,
>> > id=1032, stack(0x000000004495f000,0x0000000044a60000)]
>> > 0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10"
>> > [_thread_blocked,
>> > id=1031, stack(0x000000004485e000,0x000000004495f000)]
>> > 0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
>> > id=1030, stack(0x000000004475d000,0x000000004485e000)]
>> > 0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
>> > id=1029, stack(0x000000004465c000,0x000000004475d000)]
>> > 0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
>> > id=1028, stack(0x000000004455b000,0x000000004465c000)]
>> > 0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
>> > id=1027, stack(0x000000004445a000,0x000000004455b000)]
>> > 0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
>> > id=1026, stack(0x0000000044359000,0x000000004445a000)]
>> > 0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
>> > id=1025, stack(0x0000000044258000,0x0000000044359000)]
>> > 0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
>> > id=1024, stack(0x0000000044157000,0x0000000044258000)]
>> > 0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
>> > id=1023, stack(0x0000000044056000,0x0000000044157000)]
>> > 0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
>> > id=1022, stack(0x0000000043f55000,0x0000000044056000)]
>> > 0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER"
>> > [_thread_blocked,
>> > id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
>> > 0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon
>> > [_thread_blocked,
>> > id=1017, stack(0x0000000041499000,0x000000004159a000)]
>> > 0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked,
>> > id=1009,
>> > stack(0x0000000043c52000,0x0000000043d53000)]
>> > 0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1"
>> > [_thread_blocked,
>> > id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
>> > 0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
>> > [_thread_blocked, id=1007, stack(0x0000000043a50000,0x0000000043b51000)]
>> > 0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
>> > [_thread_blocked, id=1006, stack(0x000000004394f000,0x0000000043a50000)]
>> > 0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
>> > [_thread_blocked, id=1005, stack(0x000000004384e000,0x000000004394f000)]
>> > 0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4"
>> > [_thread_blocked,
>> > id=1004, stack(0x000000004374d000,0x000000004384e000)]
>> > 0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3"
>> > [_thread_blocked,
>> > id=1003, stack(0x000000004364c000,0x000000004374d000)]
>> > 0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2"
>> > [_thread_blocked,
>> > id=1002, stack(0x000000004354b000,0x000000004364c000)]
>> > 0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1"
>> > [_thread_blocked,
>> > id=1001, stack(0x000000004344a000,0x000000004354b000)]
>> > 0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1"
>> > [_thread_blocked,
>> > id=1000, stack(0x0000000043349000,0x000000004344a000)]
>> > 0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
>> > id=999, stack(0x0000000043248000,0x0000000043349000)]
>> > 0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
>> > id=998, stack(0x0000000043147000,0x0000000043248000)]
>> > 0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked, id=997,
>> > stack(0x000000004117f000,0x0000000041280000)]
>> > 0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
>> > [_thread_in_native, id=995,
>> > stack(0x0000000040f7d000,0x000000004107e000)]
>> > 0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
>> > [_thread_in_native, id=994,
>> > stack(0x0000000043046000,0x0000000043147000)]
>> > 0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
>> > [_thread_in_native, id=989,
>> > stack(0x0000000040a48000,0x0000000040b49000)]
>> > 0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
>> > [_thread_blocked, id=988, stack(0x0000000042f45000,0x0000000043046000)]
>> > 0x00002aabbc024000 JavaThread "CompilerThread1" daemon
>> > [_thread_blocked,
>> > id=987, stack(0x0000000042e44000,0x0000000042f45000)]
>> > 0x00002aabbc022000 JavaThread "CompilerThread0" daemon
>> > [_thread_blocked,
>> > id=986, stack(0x0000000042d43000,0x0000000042e44000)]
>> > 0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon
>> > [_thread_blocked,
>> > id=985, stack(0x00000000417e8000,0x00000000418e9000)]
>> > 0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
>> > [_thread_blocked, id=984, stack(0x00000000416e7000,0x00000000417e8000)]
>> > 0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked,
>> > id=983,
>> > stack(0x0000000042c42000,0x0000000042d43000)]
>> > 0x0000000047424800 JavaThread "Reference Handler" daemon
>> > [_thread_blocked,
>> > id=982, stack(0x0000000042b41000,0x0000000042c42000)]
>> > 0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
>> > stack(0x00000000412b1000,0x00000000413b2000)]
>> > Other Threads:
>> > 0x000000004741d800 VMThread [stack:
>> > 0x0000000042a40000,0x0000000042b41000]
>> > [id=981]
>> > 0x00002aabbc181000 WatcherThread [stack:
>> > 0x000000004107e000,0x000000004117f000] [id=996]
>> > =>0x0000000047276800 (exited) GCTaskThread [stack:
>> > 0x0000000000000000,0x0000000000000000] [id=974]
>> > VM state:at safepoint (normal execution)
>> > VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
>> > [0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
>> > [0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
>> > Heap
>> > par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
>> > 0x00002aaabf050000, 0x00002aaabf050000)
>> > eden space 272640K, 100% used [0x00002aaaae1f0000, 0x00002aaabec30000,
>> > 0x00002aaabec30000)
>> > from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
>> > 0x00002aaabf050000)
>> > to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
>> > 0x00002aaabee40000)
>> > concurrent mark-sweep generation total 3184972K, used 2657590K
>> > [0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
>> > concurrent-mark-sweep perm gen total 26272K, used 15697K
>> > [0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
>> > VM Arguments:
>> > jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90
>> > -XX:+AggressiveOpts
>> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> > -XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
>> > -XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
>> > -Dcom.sun.management.jmxremote.ssl=false
>> > -Dcom.sun.management.jmxremote.authenticate=false
>> > -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
>> > -Dcassandra-pidfile=/var/run/cassandra.pid
>> > java_command: org.apache.cassandra.thrift.CassandraDaemon
>> > Launcher Type: SUN_STANDARD
>> > Environment Variables:
>> > JAVA_HOME=/usr/java/jdk1.6.0_17/
>> >
>> > PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
>> >
>> > LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
>> > SHELL=/bin/bash
>> > Signal Handlers:
>> > SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff,
>> > sa_flags=0x10000004
>> > SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> > SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> > SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
>> > sa_flags=0x10000004
>> > SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff,
>> > sa_flags=0x10000004
>> > SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
>> > SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000,
>> > sa_flags=0x10000004
>> > SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>> > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
>> > SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff,
>> > sa_flags=0x10000004
>> > SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff,
>> > sa_flags=0x10000004
>> >
>> > --------------- S Y S T E M ---------------
>> > OS:CentOS release 5.4 (Final)
>> > uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010
>> > x86_64
>> > libc:glibc 2.5 NPTL 2.5
>> > rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS infinity
>> > load average:2.50 3.93 5.66
>> > CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
>> > stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1,
>> > sse4.2, ht
>> > Memory: 4k page, physical 16429856k(55128k free), swap 5242872k(5220216k
>> > free)
>> > vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64
>> > JRE
>> > (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc
>> > 3.2.2
>> > (SuSE Linux)
>> > time: Tue May 4 17:03:44 2010
>> > elapsed time: 11909 seconds
>> >
>> >
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: cassandra jvm crash in GCTaskThread
Posted by Ran Tavory <ra...@gmail.com>.
has anyone tried jvm _20 or _19 with cassandra?
On Wed, May 5, 2010 at 10:00 AM, Nathan McCall <na...@vervewireless.com>wrote:
> Ran,
> You may want to upgrade your jvm. I had a similar core with a heavily
> loaded tomcat recently and found the following on the release notes
> page for 1.6.0_18:
> "Card-Marking Optimization Issue
>
> A flaw in the implementation of a card-marking performance
> optimization in the JVM can cause heap corruption under some
> circumstances. This issue affects the CMS garbage collector prior to
> 6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The
> serial garbage collector is not affected. Applications most likely to
> be affected by this issue are those that allocate very large objects
> which would not normally fit in Eden, or those that make extensive use
> of JNI Critical Sections (JNI Get/Release*Critical)."
>
> http://java.sun.com/javase/6/webnotes/6u18.html
>
> Note that there is an apparent work-around jvm option to add:
> -XX:-ReduceInitialCardMarks
>
> -Nate
>
>
> On Tue, May 4, 2010 at 11:24 PM, Ran Tavory <ra...@gmail.com> wrote:
> > Running a cluster of 0.6.1, one of the hosts crashed during GC.
> > Can anyone suggest what needs tweaking to prevent that?
> > The last configuration change I made was to
> > change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I
> can't
> > determine if it's related.
> >
> > System is 64 bit centos, the rest of the system info is at the bottom of
> the
> > email.
> > The last lines before the crash from system.log are:
> >
> > INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913 CompactionManager.java
> > (line 246) Compacting
> >
> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
> > INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line
> 407)
> > Discarding obsolete commit
> >
> log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
> > INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820 CompactionManager.java
> > (line 326) Compacted to
> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
> > 291072443/291071793 bytes for 123292 keys. Time: 19906ms.
> > And a summary of hs_err is here. The rest is attached. Let me know if
> you've
> > see this before and can advice, thanks!
> > #
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > # SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
> > #
> > # JRE version: 6.0_17-b04
> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
> > linux-amd64 )
> > # Problematic frame:
> > # V [libjvm.so+0x1ffb76]
> > #
> > # If you would like to submit a bug report, please visit:
> > # http://java.sun.com/webapps/bugreport/crash.jsp
> > #
> > --------------- T H R E A D ---------------
> > Current thread (0x0000000047276800): GCTaskThread [stack:
> > 0x0000000000000000,0x0000000000000000] [id=974]
> > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
> > si_addr=0x0000000000000000
> >
> > --------------- P R O C E S S ---------------
> > Java Threads: ( => current thread )
> > 0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1"
> [_thread_blocked,
> > id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
> > 0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
> > [_thread_blocked, id=1270, stack(0x0000000049dbf000,0x0000000049ec0000)]
> > 0x0000000047691000 JavaThread "Thread-26" [_thread_in_native, id=1191,
> > stack(0x0000000043d53000,0x0000000043e54000)]
> > 0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native, id=1117,
> > stack(0x0000000048daf000,0x0000000048eb0000)]
> > 0x0000000047a42800 JavaThread "WRITE-/192.168.252.61"
> [_thread_blocked,
> > id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
> > 0x0000000047abc800 JavaThread "WRITE-/192.168.252.62"
> [_thread_blocked,
> > id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
> > 0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked,
> id=1108,
> > stack(0x0000000049abc000,0x0000000049bbd000)]
> > 0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked,
> > id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
> > 0x0000000047693800 JavaThread "Thread-16" [_thread_in_native, id=1106,
> > stack(0x00000000498ba000,0x00000000499bb000)]
> > 0x0000000047693000 JavaThread "WRITE-/192.168.252.99"
> [_thread_blocked,
> > id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
> > 0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
> > stack(0x00000000496b8000,0x00000000497b9000)]
> > 0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native, id=1103,
> > stack(0x00000000495b7000,0x00000000496b8000)]
> > 0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59"
> [_thread_blocked,
> > id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
> > 0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native, id=1101,
> > stack(0x00000000493b5000,0x00000000494b6000)]
> > 0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61"
> [_thread_blocked,
> > id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
> > 0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58"
> [_thread_blocked,
> > id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
> > 0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62"
> [_thread_blocked,
> > id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
> > 0x00000000475db000 JavaThread "Thread-12" [_thread_in_native, id=1097,
> > stack(0x0000000048fb1000,0x00000000490b2000)]
> > 0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
> > stack(0x0000000048eb0000,0x0000000048fb1000)]
> > 0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
> > stack(0x0000000048bad000,0x0000000048cae000)]
> > 0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
> > stack(0x0000000048aac000,0x0000000048bad000)]
> > 0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked, id=1091,
> > stack(0x00000000489ab000,0x0000000048aac000)]
> > 0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked,
> id=1090,
> > stack(0x00000000488aa000,0x00000000489ab000)]
> > 0x00002aabbc1ac000 JavaThread
> > "ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
> > id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
> > 0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked,
> id=1088,
> > stack(0x0000000040694000,0x0000000040795000)]
> > 0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
> > id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
> > 0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
> > [_thread_blocked, id=1086, stack(0x00000000485a7000,0x00000000486a8000)]
> > 0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
> > id=1085, stack(0x0000000040830000,0x0000000040931000)]
> > 0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked,
> > id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
> > 0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
> > id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
> > 0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
> > stack(0x00000000482a4000,0x00000000483a5000)]
> > 0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked,
> id=1078,
> > stack(0x00000000481a3000,0x00000000482a4000)]
> > 0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
> > id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
> > 0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
> > id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
> > 0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
> > id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
> > 0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
> > id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
> > 0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
> > id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
> > 0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
> > id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
> > 0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
> > id=1071, stack(0x0000000047086000,0x0000000047187000)]
> > 0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
> > id=1070, stack(0x0000000046f85000,0x0000000047086000)]
> > 0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
> > id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
> > 0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
> > id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
> > 0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
> > id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
> > 0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
> > id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
> > 0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
> > id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
> > 0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
> > id=1064, stack(0x000000004697f000,0x0000000046a80000)]
> > 0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
> > id=1063, stack(0x000000004687e000,0x000000004697f000)]
> > 0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
> > id=1062, stack(0x000000004677d000,0x000000004687e000)]
> > 0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
> > id=1061, stack(0x000000004667c000,0x000000004677d000)]
> > 0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
> > id=1060, stack(0x000000004657b000,0x000000004667c000)]
> > 0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
> > id=1059, stack(0x000000004647a000,0x000000004657b000)]
> > 0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
> > id=1058, stack(0x0000000046379000,0x000000004647a000)]
> > 0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
> > id=1057, stack(0x0000000046278000,0x0000000046379000)]
> > 0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
> > id=1056, stack(0x0000000046177000,0x0000000046278000)]
> > 0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
> > id=1055, stack(0x0000000046076000,0x0000000046177000)]
> > 0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
> > id=1054, stack(0x0000000045f75000,0x0000000046076000)]
> > 0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked,
> > id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
> > 0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked,
> > id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
> > 0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked,
> > id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
> > 0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked,
> > id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
> > 0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked,
> > id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
> > 0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked,
> > id=1048, stack(0x000000004596f000,0x0000000045a70000)]
> > 0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked,
> > id=1047, stack(0x000000004586e000,0x000000004596f000)]
> > 0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked,
> > id=1046, stack(0x000000004576d000,0x000000004586e000)]
> > 0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked,
> > id=1045, stack(0x000000004566c000,0x000000004576d000)]
> > 0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked,
> > id=1044, stack(0x000000004556b000,0x000000004566c000)]
> > 0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked,
> > id=1043, stack(0x000000004546a000,0x000000004556b000)]
> > 0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked,
> > id=1042, stack(0x0000000045369000,0x000000004546a000)]
> > 0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked,
> > id=1041, stack(0x0000000045268000,0x0000000045369000)]
> > 0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked,
> > id=1040, stack(0x0000000045167000,0x0000000045268000)]
> > 0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked,
> > id=1039, stack(0x0000000045066000,0x0000000045167000)]
> > 0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked,
> > id=1038, stack(0x0000000044f65000,0x0000000045066000)]
> > 0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked,
> > id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
> > 0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked,
> > id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
> > 0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked,
> > id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
> > 0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked,
> > id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
> > 0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked,
> > id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
> > 0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked,
> > id=1032, stack(0x000000004495f000,0x0000000044a60000)]
> > 0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked,
> > id=1031, stack(0x000000004485e000,0x000000004495f000)]
> > 0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
> > id=1030, stack(0x000000004475d000,0x000000004485e000)]
> > 0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
> > id=1029, stack(0x000000004465c000,0x000000004475d000)]
> > 0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
> > id=1028, stack(0x000000004455b000,0x000000004465c000)]
> > 0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
> > id=1027, stack(0x000000004445a000,0x000000004455b000)]
> > 0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
> > id=1026, stack(0x0000000044359000,0x000000004445a000)]
> > 0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
> > id=1025, stack(0x0000000044258000,0x0000000044359000)]
> > 0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
> > id=1024, stack(0x0000000044157000,0x0000000044258000)]
> > 0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
> > id=1023, stack(0x0000000044056000,0x0000000044157000)]
> > 0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
> > id=1022, stack(0x0000000043f55000,0x0000000044056000)]
> > 0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked,
> > id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
> > 0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon
> [_thread_blocked,
> > id=1017, stack(0x0000000041499000,0x000000004159a000)]
> > 0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked,
> id=1009,
> > stack(0x0000000043c52000,0x0000000043d53000)]
> > 0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1"
> [_thread_blocked,
> > id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
> > 0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
> > [_thread_blocked, id=1007, stack(0x0000000043a50000,0x0000000043b51000)]
> > 0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
> > [_thread_blocked, id=1006, stack(0x000000004394f000,0x0000000043a50000)]
> > 0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
> > [_thread_blocked, id=1005, stack(0x000000004384e000,0x000000004394f000)]
> > 0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked,
> > id=1004, stack(0x000000004374d000,0x000000004384e000)]
> > 0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked,
> > id=1003, stack(0x000000004364c000,0x000000004374d000)]
> > 0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked,
> > id=1002, stack(0x000000004354b000,0x000000004364c000)]
> > 0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked,
> > id=1001, stack(0x000000004344a000,0x000000004354b000)]
> > 0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1"
> [_thread_blocked,
> > id=1000, stack(0x0000000043349000,0x000000004344a000)]
> > 0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
> > id=999, stack(0x0000000043248000,0x0000000043349000)]
> > 0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
> > id=998, stack(0x0000000043147000,0x0000000043248000)]
> > 0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked, id=997,
> > stack(0x000000004117f000,0x0000000041280000)]
> > 0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
> > [_thread_in_native, id=995, stack(0x0000000040f7d000,0x000000004107e000)]
> > 0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
> > [_thread_in_native, id=994, stack(0x0000000043046000,0x0000000043147000)]
> > 0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
> > [_thread_in_native, id=989, stack(0x0000000040a48000,0x0000000040b49000)]
> > 0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
> > [_thread_blocked, id=988, stack(0x0000000042f45000,0x0000000043046000)]
> > 0x00002aabbc024000 JavaThread "CompilerThread1" daemon
> [_thread_blocked,
> > id=987, stack(0x0000000042e44000,0x0000000042f45000)]
> > 0x00002aabbc022000 JavaThread "CompilerThread0" daemon
> [_thread_blocked,
> > id=986, stack(0x0000000042d43000,0x0000000042e44000)]
> > 0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon
> [_thread_blocked,
> > id=985, stack(0x00000000417e8000,0x00000000418e9000)]
> > 0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
> > [_thread_blocked, id=984, stack(0x00000000416e7000,0x00000000417e8000)]
> > 0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked,
> id=983,
> > stack(0x0000000042c42000,0x0000000042d43000)]
> > 0x0000000047424800 JavaThread "Reference Handler" daemon
> [_thread_blocked,
> > id=982, stack(0x0000000042b41000,0x0000000042c42000)]
> > 0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
> > stack(0x00000000412b1000,0x00000000413b2000)]
> > Other Threads:
> > 0x000000004741d800 VMThread [stack:
> 0x0000000042a40000,0x0000000042b41000]
> > [id=981]
> > 0x00002aabbc181000 WatcherThread [stack:
> > 0x000000004107e000,0x000000004117f000] [id=996]
> > =>0x0000000047276800 (exited) GCTaskThread [stack:
> > 0x0000000000000000,0x0000000000000000] [id=974]
> > VM state:at safepoint (normal execution)
> > VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> > [0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
> > [0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
> > Heap
> > par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
> > 0x00002aaabf050000, 0x00002aaabf050000)
> > eden space 272640K, 100% used [0x00002aaaae1f0000, 0x00002aaabec30000,
> > 0x00002aaabec30000)
> > from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
> > 0x00002aaabf050000)
> > to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
> > 0x00002aaabee40000)
> > concurrent mark-sweep generation total 3184972K, used 2657590K
> > [0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
> > concurrent-mark-sweep perm gen total 26272K, used 15697K
> > [0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
> > VM Arguments:
> > jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90
> -XX:+AggressiveOpts
> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> > -XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
> > -XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
> > -Dcassandra-pidfile=/var/run/cassandra.pid
> > java_command: org.apache.cassandra.thrift.CassandraDaemon
> > Launcher Type: SUN_STANDARD
> > Environment Variables:
> > JAVA_HOME=/usr/java/jdk1.6.0_17/
> >
> PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
> >
> LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
> > SHELL=/bin/bash
> > Signal Handlers:
> > SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000, sa_flags=0x10000004
> > SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> > SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> > SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> >
> > --------------- S Y S T E M ---------------
> > OS:CentOS release 5.4 (Final)
> > uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010
> x86_64
> > libc:glibc 2.5 NPTL 2.5
> > rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS infinity
> > load average:2.50 3.93 5.66
> > CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
> > stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2,
> ht
> > Memory: 4k page, physical 16429856k(55128k free), swap 5242872k(5220216k
> > free)
> > vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64 JRE
> > (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc 3.2.2
> > (SuSE Linux)
> > time: Tue May 4 17:03:44 2010
> > elapsed time: 11909 seconds
> >
> >
>
Re: cassandra jvm crash in GCTaskThread
Posted by Nathan McCall <na...@vervewireless.com>.
Ran,
You may want to upgrade your jvm. I had a similar core with a heavily
loaded tomcat recently and found the following on the release notes
page for 1.6.0_18:
"Card-Marking Optimization Issue
A flaw in the implementation of a card-marking performance
optimization in the JVM can cause heap corruption under some
circumstances. This issue affects the CMS garbage collector prior to
6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The
serial garbage collector is not affected. Applications most likely to
be affected by this issue are those that allocate very large objects
which would not normally fit in Eden, or those that make extensive use
of JNI Critical Sections (JNI Get/Release*Critical)."
http://java.sun.com/javase/6/webnotes/6u18.html
Note that there is an apparent work-around jvm option to add:
-XX:-ReduceInitialCardMarks
-Nate
On Tue, May 4, 2010 at 11:24 PM, Ran Tavory <ra...@gmail.com> wrote:
> Running a cluster of 0.6.1, one of the hosts crashed during GC.
> Can anyone suggest what needs tweaking to prevent that?
> The last configuration change I made was to
> change <DiskAccessMode>standard</DiskAccessMode> (it was auto); but I can't
> determine if it's related.
>
> System is 64 bit centos, the rest of the system info is at the bottom of the
> email.
> The last lines before the crash from system.log are:
>
> INFO [COMPACTION-POOL:1] 2010-05-04 17:01:49,913 CompactionManager.java
> (line 246) Compacting
> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-607-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-608-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-609-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-610-Data.db')]
> INFO [COMMIT-LOG-WRITER] 2010-05-04 17:01:49,920 CommitLog.java (line 407)
> Discarding obsolete commit
> log:CommitLogSegment(/outbrain/cassandra/commitlog/CommitLog-1273006181463.log)
> INFO [COMPACTION-POOL:1] 2010-05-04 17:02:09,820 CompactionManager.java
> (line 326) Compacted to
> /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-611-Data.db.
> 291072443/291071793 bytes for 123292 keys. Time: 19906ms.
> And a summary of hs_err is here. The rest is attached. Let me know if you've
> see this before and can advice, thanks!
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00002add529adb76, pid=943, tid=1112779072
> #
> # JRE version: 6.0_17-b04
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode
> linux-amd64 )
> # Problematic frame:
> # V [libjvm.so+0x1ffb76]
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
> --------------- T H R E A D ---------------
> Current thread (0x0000000047276800): GCTaskThread [stack:
> 0x0000000000000000,0x0000000000000000] [id=974]
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (),
> si_addr=0x0000000000000000
>
> --------------- P R O C E S S ---------------
> Java Threads: ( => current thread )
> 0x0000000047a08800 JavaThread "FILEUTILS-DELETE-POOL:1" [_thread_blocked,
> id=1297, stack(0x0000000043e54000,0x0000000043f55000)]
> 0x0000000047ab9000 JavaThread "CACHETABLE-TIMER-3" daemon
> [_thread_blocked, id=1270, stack(0x0000000049dbf000,0x0000000049ec0000)]
> 0x0000000047691000 JavaThread "Thread-26" [_thread_in_native, id=1191,
> stack(0x0000000043d53000,0x0000000043e54000)]
> 0x00000000476ac000 JavaThread "Thread-19" [_thread_in_native, id=1117,
> stack(0x0000000048daf000,0x0000000048eb0000)]
> 0x0000000047a42800 JavaThread "WRITE-/192.168.252.61" [_thread_blocked,
> id=1110, stack(0x0000000049cbe000,0x0000000049dbf000)]
> 0x0000000047abc800 JavaThread "WRITE-/192.168.252.62" [_thread_blocked,
> id=1109, stack(0x0000000049bbd000,0x0000000049cbe000)]
> 0x00002aabc841b800 JavaThread "Hint delivery" [_thread_blocked, id=1108,
> stack(0x0000000049abc000,0x0000000049bbd000)]
> 0x00002aabc841a800 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked,
> id=1107, stack(0x00000000499bb000,0x0000000049abc000)]
> 0x0000000047693800 JavaThread "Thread-16" [_thread_in_native, id=1106,
> stack(0x00000000498ba000,0x00000000499bb000)]
> 0x0000000047693000 JavaThread "WRITE-/192.168.252.99" [_thread_blocked,
> id=1105, stack(0x00000000497b9000,0x00000000498ba000)]
> 0x0000000047b05000 JavaThread "Thread-15" [_thread_blocked, id=1104,
> stack(0x00000000496b8000,0x00000000497b9000)]
> 0x0000000047a7b000 JavaThread "Thread-14" [_thread_in_native, id=1103,
> stack(0x00000000495b7000,0x00000000496b8000)]
> 0x00002aabc8056000 JavaThread "WRITE-/192.168.254.59" [_thread_blocked,
> id=1102, stack(0x00000000494b6000,0x00000000495b7000)]
> 0x00002aabccf61800 JavaThread "Thread-13" [_thread_in_native, id=1101,
> stack(0x00000000493b5000,0x00000000494b6000)]
> 0x00002aabccf5f000 JavaThread "WRITE-/192.168.252.61" [_thread_blocked,
> id=1100, stack(0x00000000492b4000,0x00000000493b5000)]
> 0x00002aabc8054000 JavaThread "WRITE-/192.168.254.58" [_thread_blocked,
> id=1099, stack(0x00000000491b3000,0x00000000492b4000)]
> 0x00002aabc8053800 JavaThread "WRITE-/192.168.252.62" [_thread_blocked,
> id=1098, stack(0x00000000490b2000,0x00000000491b3000)]
> 0x00000000475db000 JavaThread "Thread-12" [_thread_in_native, id=1097,
> stack(0x0000000048fb1000,0x00000000490b2000)]
> 0x00000000475d8000 JavaThread "Thread-11" [_thread_blocked, id=1096,
> stack(0x0000000048eb0000,0x0000000048fb1000)]
> 0x00002aabbc2ac800 JavaThread "Timer-1" [_thread_blocked, id=1093,
> stack(0x0000000048bad000,0x0000000048cae000)]
> 0x00002aabbc1b0800 JavaThread "Timer-0" [_thread_blocked, id=1092,
> stack(0x0000000048aac000,0x0000000048bad000)]
> 0x00002aabbc1ae800 JavaThread "LB-TARGET:1" [_thread_blocked, id=1091,
> stack(0x00000000489ab000,0x0000000048aac000)]
> 0x00002aabbc1ad000 JavaThread "LB-OPERATIONS:1" [_thread_blocked, id=1090,
> stack(0x00000000488aa000,0x00000000489ab000)]
> 0x00002aabbc1ac000 JavaThread
> "ACCEPT-ob2057.ladc1.outbrain.com/192.168.254.57" [_thread_in_native,
> id=1089, stack(0x00000000487a9000,0x00000000488aa000)]
> 0x00002aabbc1ab800 JavaThread "GC inspection" [_thread_blocked, id=1088,
> stack(0x0000000040694000,0x0000000040795000)]
> 0x00002aabcc1fd800 JavaThread "COMPACTION-POOL:1" [_thread_blocked,
> id=1087, stack(0x00000000486a8000,0x00000000487a9000)]
> 0x00002aabbc29f000 JavaThread "PERIODIC-COMMIT-LOG-SYNCER"
> [_thread_blocked, id=1086, stack(0x00000000485a7000,0x00000000486a8000)]
> 0x00002aabbc29a000 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
> id=1085, stack(0x0000000040830000,0x0000000040931000)]
> 0x00002aabbc194800 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked,
> id=1082, stack(0x00000000484a6000,0x00000000485a7000)]
> 0x00002aabbc275000 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
> id=1081, stack(0x00000000483a5000,0x00000000484a6000)]
> 0x00002aabbc273000 JavaThread "GMFD:1" [_thread_blocked, id=1079,
> stack(0x00000000482a4000,0x00000000483a5000)]
> 0x00002aabbc271000 JavaThread "STREAM-STAGE:1" [_thread_blocked, id=1078,
> stack(0x00000000481a3000,0x00000000482a4000)]
> 0x00002aabbc26f800 JavaThread "RESPONSE-STAGE:16" [_thread_blocked,
> id=1077, stack(0x00000000480a2000,0x00000000481a3000)]
> 0x00002aabbc26d800 JavaThread "RESPONSE-STAGE:15" [_thread_blocked,
> id=1076, stack(0x0000000047fa1000,0x00000000480a2000)]
> 0x00002aabbc26b800 JavaThread "RESPONSE-STAGE:14" [_thread_blocked,
> id=1075, stack(0x0000000047ea0000,0x0000000047fa1000)]
> 0x00002aabbc269800 JavaThread "RESPONSE-STAGE:13" [_thread_blocked,
> id=1074, stack(0x0000000047d9f000,0x0000000047ea0000)]
> 0x00002aabbc267800 JavaThread "RESPONSE-STAGE:12" [_thread_blocked,
> id=1073, stack(0x0000000047c9e000,0x0000000047d9f000)]
> 0x00002aabbc265800 JavaThread "RESPONSE-STAGE:11" [_thread_blocked,
> id=1072, stack(0x0000000047b9d000,0x0000000047c9e000)]
> 0x00002aabbc263800 JavaThread "RESPONSE-STAGE:10" [_thread_blocked,
> id=1071, stack(0x0000000047086000,0x0000000047187000)]
> 0x00002aabbc261800 JavaThread "RESPONSE-STAGE:9" [_thread_blocked,
> id=1070, stack(0x0000000046f85000,0x0000000047086000)]
> 0x00002aabbc25f800 JavaThread "RESPONSE-STAGE:8" [_thread_blocked,
> id=1069, stack(0x0000000046e84000,0x0000000046f85000)]
> 0x00002aabbc25d800 JavaThread "RESPONSE-STAGE:7" [_thread_blocked,
> id=1068, stack(0x0000000046d83000,0x0000000046e84000)]
> 0x00002aabbc25c000 JavaThread "RESPONSE-STAGE:6" [_thread_blocked,
> id=1067, stack(0x0000000046c82000,0x0000000046d83000)]
> 0x00002aabbc20b000 JavaThread "RESPONSE-STAGE:5" [_thread_blocked,
> id=1066, stack(0x0000000046b81000,0x0000000046c82000)]
> 0x00002aabbc209800 JavaThread "RESPONSE-STAGE:4" [_thread_blocked,
> id=1065, stack(0x0000000046a80000,0x0000000046b81000)]
> 0x00002aabbc207800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked,
> id=1064, stack(0x000000004697f000,0x0000000046a80000)]
> 0x00002aabbc205800 JavaThread "RESPONSE-STAGE:2" [_thread_blocked,
> id=1063, stack(0x000000004687e000,0x000000004697f000)]
> 0x00002aabbc203800 JavaThread "RESPONSE-STAGE:1" [_thread_blocked,
> id=1062, stack(0x000000004677d000,0x000000004687e000)]
> 0x00002aabbc201800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked,
> id=1061, stack(0x000000004667c000,0x000000004677d000)]
> 0x00002aabbc1ff800 JavaThread "ROW-READ-STAGE:7" [_thread_blocked,
> id=1060, stack(0x000000004657b000,0x000000004667c000)]
> 0x00002aabbc1fd800 JavaThread "ROW-READ-STAGE:6" [_thread_blocked,
> id=1059, stack(0x000000004647a000,0x000000004657b000)]
> 0x00002aabbc1fb800 JavaThread "ROW-READ-STAGE:5" [_thread_in_native,
> id=1058, stack(0x0000000046379000,0x000000004647a000)]
> 0x00002aabbc1f9800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked,
> id=1057, stack(0x0000000046278000,0x0000000046379000)]
> 0x00002aabbc1f7800 JavaThread "ROW-READ-STAGE:3" [_thread_in_native,
> id=1056, stack(0x0000000046177000,0x0000000046278000)]
> 0x00002aabbc1f5800 JavaThread "ROW-READ-STAGE:2" [_thread_in_native,
> id=1055, stack(0x0000000046076000,0x0000000046177000)]
> 0x00002aabbc28e800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked,
> id=1054, stack(0x0000000045f75000,0x0000000046076000)]
> 0x00002aabbc28c800 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked,
> id=1053, stack(0x0000000045e74000,0x0000000045f75000)]
> 0x00002aabbc28a800 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked,
> id=1052, stack(0x0000000045d73000,0x0000000045e74000)]
> 0x00002aabbc288800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked,
> id=1051, stack(0x0000000045c72000,0x0000000045d73000)]
> 0x00002aabbc286800 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked,
> id=1050, stack(0x0000000045b71000,0x0000000045c72000)]
> 0x00002aabbc284800 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked,
> id=1049, stack(0x0000000045a70000,0x0000000045b71000)]
> 0x00002aabbc282800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked,
> id=1048, stack(0x000000004596f000,0x0000000045a70000)]
> 0x00002aabbc280800 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked,
> id=1047, stack(0x000000004586e000,0x000000004596f000)]
> 0x00002aabbc27e800 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked,
> id=1046, stack(0x000000004576d000,0x000000004586e000)]
> 0x00002aabbc27c800 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked,
> id=1045, stack(0x000000004566c000,0x000000004576d000)]
> 0x00002aabbc27a800 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked,
> id=1044, stack(0x000000004556b000,0x000000004566c000)]
> 0x00002aabbc279000 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked,
> id=1043, stack(0x000000004546a000,0x000000004556b000)]
> 0x00002aabbc220000 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked,
> id=1042, stack(0x0000000045369000,0x000000004546a000)]
> 0x00002aabbc21e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked,
> id=1041, stack(0x0000000045268000,0x0000000045369000)]
> 0x00002aabbc21c000 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked,
> id=1040, stack(0x0000000045167000,0x0000000045268000)]
> 0x00002aabbc21a000 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked,
> id=1039, stack(0x0000000045066000,0x0000000045167000)]
> 0x00002aabbc218000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked,
> id=1038, stack(0x0000000044f65000,0x0000000045066000)]
> 0x00002aabbc216000 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked,
> id=1037, stack(0x0000000044e64000,0x0000000044f65000)]
> 0x00002aabbc214000 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked,
> id=1036, stack(0x0000000044d63000,0x0000000044e64000)]
> 0x00002aabbc212000 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked,
> id=1035, stack(0x0000000044c62000,0x0000000044d63000)]
> 0x00002aabbc210800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked,
> id=1034, stack(0x0000000044b61000,0x0000000044c62000)]
> 0x00002aabbc259000 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked,
> id=1033, stack(0x0000000044a60000,0x0000000044b61000)]
> 0x00002aabbc257000 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked,
> id=1032, stack(0x000000004495f000,0x0000000044a60000)]
> 0x00002aabbc255000 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked,
> id=1031, stack(0x000000004485e000,0x000000004495f000)]
> 0x00002aabbc253000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
> id=1030, stack(0x000000004475d000,0x000000004485e000)]
> 0x00002aabbc040800 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
> id=1029, stack(0x000000004465c000,0x000000004475d000)]
> 0x00002aabbc03e800 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
> id=1028, stack(0x000000004455b000,0x000000004465c000)]
> 0x00002aabbc03c800 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
> id=1027, stack(0x000000004445a000,0x000000004455b000)]
> 0x00002aabbc03a800 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
> id=1026, stack(0x0000000044359000,0x000000004445a000)]
> 0x00002aabbc036000 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
> id=1025, stack(0x0000000044258000,0x0000000044359000)]
> 0x00002aabbc034800 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
> id=1024, stack(0x0000000044157000,0x0000000044258000)]
> 0x00002aabbc033000 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
> id=1023, stack(0x0000000044056000,0x0000000044157000)]
> 0x00002aabbc24d800 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
> id=1022, stack(0x0000000043f55000,0x0000000044056000)]
> 0x00002aabbc24d000 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked,
> id=1021, stack(0x0000000040c68000,0x0000000040d69000)]
> 0x00000000475e5000 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked,
> id=1017, stack(0x0000000041499000,0x000000004159a000)]
> 0x00002aabbc245800 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=1009,
> stack(0x0000000043c52000,0x0000000043d53000)]
> 0x00002aabbc243000 JavaThread "MESSAGE-STREAMING-POOL:1" [_thread_blocked,
> id=1008, stack(0x0000000043b51000,0x0000000043c52000)]
> 0x00002aabbc241800 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
> [_thread_blocked, id=1007, stack(0x0000000043a50000,0x0000000043b51000)]
> 0x00002aabbc23d800 JavaThread "CACHETABLE-TIMER-2" daemon
> [_thread_blocked, id=1006, stack(0x000000004394f000,0x0000000043a50000)]
> 0x00002aabbc23f000 JavaThread "CACHETABLE-TIMER-1" daemon
> [_thread_blocked, id=1005, stack(0x000000004384e000,0x000000004394f000)]
> 0x00002aabbc232000 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked,
> id=1004, stack(0x000000004374d000,0x000000004384e000)]
> 0x00002aabbc230000 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked,
> id=1003, stack(0x000000004364c000,0x000000004374d000)]
> 0x00002aabbc233000 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked,
> id=1002, stack(0x000000004354b000,0x000000004364c000)]
> 0x00002aabbc235000 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked,
> id=1001, stack(0x000000004344a000,0x000000004354b000)]
> 0x00002aabbc226800 JavaThread "MEMTABLE-POST-FLUSHER:1" [_thread_blocked,
> id=1000, stack(0x0000000043349000,0x000000004344a000)]
> 0x00002aabbc20f000 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
> id=999, stack(0x0000000043248000,0x0000000043349000)]
> 0x00002aabbc1b8800 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
> id=998, stack(0x0000000043147000,0x0000000043248000)]
> 0x00002aabbc1f3800 JavaThread "FLUSH-TIMER" [_thread_blocked, id=997,
> stack(0x000000004117f000,0x0000000041280000)]
> 0x00002aabbc17e000 JavaThread "RMI TCP Accept-0" daemon
> [_thread_in_native, id=995, stack(0x0000000040f7d000,0x000000004107e000)]
> 0x00002aabbc178800 JavaThread "RMI TCP Accept-9004" daemon
> [_thread_in_native, id=994, stack(0x0000000043046000,0x0000000043147000)]
> 0x00002aabbc169800 JavaThread "RMI TCP Accept-0" daemon
> [_thread_in_native, id=989, stack(0x0000000040a48000,0x0000000040b49000)]
> 0x00002aabbc026800 JavaThread "Low Memory Detector" daemon
> [_thread_blocked, id=988, stack(0x0000000042f45000,0x0000000043046000)]
> 0x00002aabbc024000 JavaThread "CompilerThread1" daemon [_thread_blocked,
> id=987, stack(0x0000000042e44000,0x0000000042f45000)]
> 0x00002aabbc022000 JavaThread "CompilerThread0" daemon [_thread_blocked,
> id=986, stack(0x0000000042d43000,0x0000000042e44000)]
> 0x00002aabbc01f800 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
> id=985, stack(0x00000000417e8000,0x00000000418e9000)]
> 0x00002aabbc01e000 JavaThread "Surrogate Locker Thread (CMS)" daemon
> [_thread_blocked, id=984, stack(0x00000000416e7000,0x00000000417e8000)]
> 0x000000004742c000 JavaThread "Finalizer" daemon [_thread_blocked, id=983,
> stack(0x0000000042c42000,0x0000000042d43000)]
> 0x0000000047424800 JavaThread "Reference Handler" daemon [_thread_blocked,
> id=982, stack(0x0000000042b41000,0x0000000042c42000)]
> 0x000000004725d000 JavaThread "main" [_thread_in_native, id=945,
> stack(0x00000000412b1000,0x00000000413b2000)]
> Other Threads:
> 0x000000004741d800 VMThread [stack: 0x0000000042a40000,0x0000000042b41000]
> [id=981]
> 0x00002aabbc181000 WatcherThread [stack:
> 0x000000004107e000,0x000000004117f000] [id=996]
> =>0x0000000047276800 (exited) GCTaskThread [stack:
> 0x0000000000000000,0x0000000000000000] [id=974]
> VM state:at safepoint (normal execution)
> VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
> [0x000000004725a730] Threads_lock - owner thread: 0x000000004741d800
> [0x000000004725acb0] Heap_lock - owner thread: 0x00002aabbc1fd800
> Heap
> par new generation total 274752K, used 272640K [0x00002aaaae1f0000,
> 0x00002aaabf050000, 0x00002aaabf050000)
> eden space 272640K, 100% used [0x00002aaaae1f0000, 0x00002aaabec30000,
> 0x00002aaabec30000)
> from space 2112K, 0% used [0x00002aaabee40000, 0x00002aaabee40000,
> 0x00002aaabf050000)
> to space 2112K, 0% used [0x00002aaabec30000, 0x00002aaabec30000,
> 0x00002aaabee40000)
> concurrent mark-sweep generation total 3184972K, used 2657590K
> [0x00002aaabf050000, 0x00002aab816a3000, 0x00002aabae1f0000)
> concurrent-mark-sweep perm gen total 26272K, used 15697K
> [0x00002aabae1f0000, 0x00002aabafb98000, 0x00002aabb35f0000)
> VM Arguments:
> jvm_args: -ea -Xms128M -Xmx4G -XX:TargetSurvivorRatio=90 -XX:+AggressiveOpts
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:SurvivorRatio=128
> -XX:MaxTenuringThreshold=0 -Dcom.sun.management.jmxremote.port=9004
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf
> -Dcassandra-pidfile=/var/run/cassandra.pid
> java_command: org.apache.cassandra.thrift.CassandraDaemon
> Launcher Type: SUN_STANDARD
> Environment Variables:
> JAVA_HOME=/usr/java/jdk1.6.0_17/
> PATH=/usr/kerberos/bin:/usr/java/jdk1.6.0_17//bin:/outbrain/cassandra/apache-cassandra-0.6.1/bin:/usr/local/bin:/bin:/usr/bin:/opt/dell/srvadmin/bin:/outbrain/cassandra/bin
> LD_LIBRARY_PATH=/usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64
> SHELL=/bin/bash
> Signal Handlers:
> SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000, sa_flags=0x10000004
> SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
> SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
> SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
>
> --------------- S Y S T E M ---------------
> OS:CentOS release 5.4 (Final)
> uname:Linux 2.6.18-164.15.1.el5 #1 SMP Wed Mar 17 11:30:06 EDT 2010 x86_64
> libc:glibc 2.5 NPTL 2.5
> rlimit: STACK 10240k, CORE 0k, NPROC infinity, NOFILE 63536, AS infinity
> load average:2.50 3.93 5.66
> CPU:total 16 (8 cores per cpu, 2 threads per core) family 6 model 26
> stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, ht
> Memory: 4k page, physical 16429856k(55128k free), swap 5242872k(5220216k
> free)
> vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64 JRE
> (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc 3.2.2
> (SuSE Linux)
> time: Tue May 4 17:03:44 2010
> elapsed time: 11909 seconds
>
>