You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by il-woon Ahn <pr...@gmail.com> on 2010/08/01 10:04:12 UTC
Two questions : Server crash during compaction and
UnavailableException
Hi all
Nowadays I'm using Cassandra DB under high read/write loads.
I have two questions:
First, Cassandra suddenly dies during compaction. Java core dump says that
the last thread run was "COMPACTION-POOL:1".
I suspect that my business logic could lead size of columns in a column
family per a row to be greater than two gigabytes. (but i couldn't confirm
it yet)
Can this be a cause of the server down and is there any solution? (should I
wait 0.7?)
Second, It seems that my client program often get UnavailableException from
Cassandra when Cass is running in normal.
Is it possible to get such an exception in high read/write loads even if
Cassandra is up?
I handled this to try to send client's request again after 1 sec, like
handling TimedOutException. Is there any other recommended reaction?
Thanks in advance.
Regards
Ilun
Re: Two questions : Server crash during compaction and
UnavailableException
Posted by Ilun Ahn <pr...@gmail.com>.
You're right... I missed posting crash log.
I was too busy and under press of business at that time. Please understand.
These are head and tail of the JVM crash log when it stopped :
--------------- T H R E A D ---------------
Current thread (0x0000002ca9903400): JavaThread "COMPACTION-POOL:1"
[_thread_in_vm, id=10768]
siginfo:si_signo=11, si_errno=0, si_code=128, si_addr=0x0000000000000000
Registers:
RAX=0x0000002cad287d10, RBX=0x0000002ca9903400, RCX=0x0000002cad2880f8,
RDX=0x0000002cad287d20
RSP=0x0000000042149160, RBP=0x0000000042149170, RSI=0x0000002b0b7e40c8,
RDI=0x0100048c9b42085b
R8 =0x0000002cac795910, R9 =0x0000000000002a10, R10=0x0000002a95f5d200,
R11=0x0000002cac795910
R12=0x0000002b20d2cb90, R13=0x00000000421491c0, R14=0x0000000042149268,
R15=0x0000002ca9903400
RIP=0x0000002a95c05727, EFL=0x0000000000010206, CSGSFS=0x0000000000000002,
ERR=0x0000000000000000
TRAPNO=0x000000000000000d
Top of Stack: (sp=0x0000000042149160)
0x0000000042149160: 0000002ca9903400 0000002b20d2cb90
0x0000000042149170: 0000000042149180 0000002a95ab6fcd
0x0000000042149180: 00000000421491f0 0000002a95957896
0x0000000042149190: 0000002cac795910 0000002cad287d10
0x00000000421491a0: 0000002cad287d20 0000002cad2880f8
0x00000000421491b0: 0000002ca9903400 0000000000000013
0x00000000421491c0: 0000002ca9903400 0000002ca9903400
0x00000000421491d0: 00000000421491e0 0000002b0b7e4099
0x00000000421491e0: 0000002b20d2cb90 0000002c99d6850e
0x00000000421491f0: 0000000042149248 0000002a96512bdd
0x0000000042149200: 0000002a96512b9b 0000000042149208
0x0000000042149210: 0000002c99d6850e 0000000042149268
0x0000000042149220: 0000002c9a021658 0000002c9a3e6c08
0x0000000042149230: 0000002c99d68540 0000000000000000
0x0000000042149240: 0000000042149258 00000000421492b0
0x0000000042149250: 0000002a96509528 0000000000000000
0x0000000042149260: 0000000000000000 0000000000000000
0x0000000042149270: 0000000042149270 0000002c9a1f7159
0x0000000042149280: 00000000421492c8 0000002c9a1f7518
0x0000000042149290: 0000002c9a3e7200 0000002c9a1f7168
0x00000000421492a0: 0000000042149258 00000000421492c0
0x00000000421492b0: 0000000042149310 0000002a96509528
0x00000000421492c0: 0000000000000000 0000000000000000
0x00000000421492d0: 00000000421492d0 0000002c9a0966c5
0x00000000421492e0: 0000000042149348 0000002c9a09bf90
0x00000000421492f0: 0000002c9a3e6e48 0000002c9a096718
0x0000000042149300: 00000000421492c0 0000000042149350
0x0000000042149310: 0000002a9e9fd1a8 0000002a96815d80
0x0000000042149320: 0000000000000000 0000000000000000
0x0000000042149330: 0000000000000000 0000000000000000
0x0000000042149340: 0000000000000000 0000000000000000
0x0000000042149350: 0000002ca9903400 0000002b20d2cbc0
Instructions: (pc=0x0000002a95c05727)
0x0000002a95c05717: 89 f0 eb ea 90 66 66 66 90 55 48 89 e5 41 54 53
0x0000002a95c05727: 0f b7 57 10 48 89 fb 44 8d 62 01 49 63 fc e8 c6
Stack: [0x000000004204a000,0x000000004214b000), sp=0x0000000042149160, free
space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0x592727]
V [libjvm.so+0x443fcd]
V [libjvm.so+0x2e4896]
j
org.apache.cassandra.db.marshal.BytesType.compare(Ljava/lang/Object;Ljava/lang/Object;)I+6
j
java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(Ljava/lang/Object;)I+9
j
java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Ljava/lang/Comparable;)Ljava/util/concurrent/ConcurrentSkipListMap$Node;+69
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j
org.apache.cassandra.db.marshal.BytesType.compare(Ljava/lang/Object;Ljava/lang/Object;)I+6
j
java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(Ljava/lang/Object;)I+9
j
java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Ljava/lang/Comparable;)Ljava/util/concurrent/ConcurrentSkipListMap$Node;+69
J
java.util.concurrent.ConcurrentSkipListMap.doPut(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
J
org.apache.cassandra.db.ColumnFamily.addColumn(Lorg/apache/cassandra/db/IColumn;)V
J
org.apache.cassandra.db.ColumnFamily.addAll(Lorg/apache/cassandra/db/ColumnFamily;)V
J
org.apache.cassandra.io.CompactionIterator.getReduced()Lorg/apache/cassandra/io/CompactionIterator$CompactedRow;
J org.apache.cassandra.io.CompactionIterator.getReduced()Ljava/lang/Object;
J
org.apache.cassandra.utils.ReducingIterator.computeNext()Ljava/lang/Object;
J com.google.common.collect.AbstractIterator.hasNext()Z
J
org.apache.cassandra.db.CompactionManager.doAntiCompaction(Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Collection;Ljava/util/Collection;Ljava/net/InetAddress;)Ljava/util/List;
j
org.apache.cassandra.db.CompactionManager.doCleanupCompaction(Lorg/apache/cassandra/db/ColumnFamilyStore;)V+22
j
org.apache.cassandra.db.CompactionManager.access$400(Lorg/apache/cassandra/db/CompactionManager;Lorg/apache/cassandra/db/ColumnFamilyStore;)V+2
j org.apache.cassandra.db.CompactionManager$2.call()Ljava/lang/Object;+8
J java.util.concurrent.FutureTask$Sync.innerRun()V
j java.util.concurrent.FutureTask.run()V+4
j
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+59
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000002cad6ff800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked,
id=16294]
0x0000002cad21b800 JavaThread "Thread-11" [_thread_in_native, id=13570]
0x0000002cad27dc00 JavaThread "Thread-10" [_thread_in_native, id=11221]
0x0000002caceccc00 JavaThread "WRITE-/10.25.32.37" [_thread_blocked,
id=10830]
0x0000002cad33c000 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked,
id=10829]
0x0000002cacd55c00 JavaThread "WRITE-/10.25.32.36" [_thread_blocked,
id=10828]
0x0000002cacd54c00 JavaThread "WRITE-/10.25.32.34" [_thread_blocked,
id=10827]
0x0000002cad2b5400 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked,
id=10826]
0x0000002cad2b3c00 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked,
id=10825]
0x0000002cad2b2800 JavaThread "GMFD:1" [_thread_blocked, id=10824]
0x0000002cad2b1000 JavaThread "STREAM-STAGE:1" [_thread_blocked, id=10823]
0x0000002cad2afc00 JavaThread "RESPONSE-STAGE:4" [_thread_blocked, id=10822]
0x0000002cad2ae800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked, id=10821]
0x0000002cad2ad000 JavaThread "RESPONSE-STAGE:2" [_thread_blocked, id=10820]
0x0000002cad2abc00 JavaThread "RESPONSE-STAGE:1" [_thread_blocked, id=10819]
0x0000002cad2aa800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked, id=10818]
0x0000002cad2a9c00 JavaThread "ROW-READ-STAGE:7" [_thread_blocked, id=10817]
0x0000002cad2a8400 JavaThread "ROW-READ-STAGE:6" [_thread_blocked, id=10816]
0x0000002cad2a6c00 JavaThread "ROW-READ-STAGE:5" [_thread_blocked, id=10815]
0x0000002cad2a5800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked, id=10814]
0x0000002cacf82400 JavaThread "ROW-READ-STAGE:3" [_thread_blocked, id=10813]
0x0000002cacf80c00 JavaThread "ROW-READ-STAGE:2" [_thread_blocked, id=10812]
0x0000002cacf7f800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked, id=10811]
0x0000002cacf7e400 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked,
id=10810]
0x0000002cacf7cc00 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked,
id=10809]
0x0000002cacf7b800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked,
id=10808]
0x0000002cacf7a000 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked,
id=10807]
0x0000002cacf78c00 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked,
id=10806]
0x0000002cacf77800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked,
id=10805]
0x0000002cacf76000 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked,
id=10804]
0x0000002cacf74c00 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked,
id=10803]
0x0000002cacf73400 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked,
id=10802]
0x0000002cacf72000 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked,
id=10801]
0x0000002cacf70c00 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked,
id=10800]
0x0000002cacf6f400 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked,
id=10799]
0x0000002cacf6e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked,
id=10798]
0x0000002cacf6c800 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked,
id=10797]
0x0000002cacf6b400 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked,
id=10796]
0x0000002cacf6a000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked,
id=10795]
0x0000002cacf68800 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked,
id=10794]
0x0000002cacf67400 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked,
id=10793]
0x0000002cacf65c00 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked,
id=10792]
0x0000002cacf64800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked,
id=10791]
0x0000002cacf63400 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked,
id=10790]
0x0000002cacf61c00 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked,
id=10789]
0x0000002cacf60800 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked,
id=10788]
0x0000002cacf5f000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked,
id=10787]
0x0000002cacf5bc00 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked,
id=10786]
0x0000002cacf5a400 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked,
id=10785]
0x0000002cacf5d400 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked,
id=10784]
0x0000002cacf5c400 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked,
id=10783]
0x0000002cacf57800 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked,
id=10782]
0x0000002cad258400 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked,
id=10781]
0x0000002ca98f3400 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked,
id=10780]
0x0000002ca98f2400 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked,
id=10779]
0x0000002ca9902000 JavaThread "Thread-9" [_thread_in_native, id=10778]
0x0000002ca9901800 JavaThread "Thread-8" [_thread_in_native, id=10777]
0x0000002cad253000 JavaThread "Timer-1" [_thread_blocked, id=10776]
0x0000002cad25ec00 JavaThread "LB-TARGET:1" [_thread_blocked, id=10775]
0x0000002cad25d800 JavaThread "LB-OPERATIONS:1" [_thread_blocked, id=10774]
0x0000002cad25cc00 JavaThread "ACCEPT-krs1165.nhncorp.com/10.25.32.35"
[_thread_in_native, id=10773]
0x0000002cad25bc00 JavaThread "Timer-0" [_thread_blocked, id=10772]
0x0000002cacf19400 JavaThread "GC inspection" [_thread_blocked, id=10771]
0x0000002cad28bc00 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked,
id=10770]
0x0000002cad2a4400 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked,
id=10769]
=>0x0000002ca9903400 JavaThread "COMPACTION-POOL:1" [_thread_in_vm,
id=10768]
0x0000002cad2a3400 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked,
id=10378]
0x0000002cace74c00 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=10377]
0x0000002cad227000 JavaThread "MESSAGE-STREAMING-POOL:1" [_thread_blocked,
id=10376]
0x0000002cad225c00 JavaThread "MESSAGE-DESERIALIZER-POOL:1"
[_thread_blocked, id=10375]
0x0000002cad224800 JavaThread "CACHETABLE-TIMER-2" daemon [_thread_blocked,
id=10374]
0x0000002cad224000 JavaThread "CACHETABLE-TIMER-1" daemon [_thread_blocked,
id=10373]
0x0000002cacf55c00 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked,
id=10372]
0x0000002cacfa1800 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked,
id=10371]
0x0000002cacf9cc00 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked,
id=10370]
0x0000002cacfa2800 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked,
id=10369]
0x0000002cacf9b000 JavaThread "MEMTABLE-POST-FLUSHER:1" [_thread_blocked,
id=10368]
0x0000002cad230800 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked,
id=10367]
0x0000002cad237c00 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked,
id=10366]
0x0000002cad230400 JavaThread "FLUSH-TIMER" [_thread_blocked, id=10365]
0x0000002cacda6800 JavaThread "RMI TCP Accept-8080" daemon
[_thread_in_native, id=10363]
0x0000002cacd48800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native,
id=10362]
0x0000002ca9893400 JavaThread "Low Memory Detector" daemon [_thread_blocked,
id=10361]
0x0000002ca9891400 JavaThread "CompilerThread1" daemon [_thread_blocked,
id=10360]
0x0000002ca988f800 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=10359]
0x0000002ca988e000 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
id=10358]
0x0000002ca988c800 JavaThread "Surrogate Locker Thread (CMS)" daemon
[_thread_blocked, id=10357]
0x0000002ca9869c00 JavaThread "Finalizer" daemon [_thread_blocked, id=10356]
0x0000002ca9869000 JavaThread "Reference Handler" daemon [_thread_blocked,
id=10355]
0x0000000040112800 JavaThread "main" [_thread_in_native, id=10348]
Other Threads:
0x0000002ca9864400 VMThread [id=10354]
0x0000002cacd8d800 WatcherThread [id=10364]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
par new generation total 76672K, used 55139K [0x0000002a995d0000,
0x0000002a9e900000, 0x0000002a9e900000)
eden space 68160K, 68% used [0x0000002a995d0000, 0x0000002a9c359758,
0x0000002a9d860000)
from space 8512K, 99% used [0x0000002a9e0b0000, 0x0000002a9e8ff820,
0x0000002a9e900000)
to space 8512K, 0% used [0x0000002a9d860000, 0x0000002a9d860000,
0x0000002a9e0b0000)
concurrent mark-sweep generation total 5810916K, used 4774431K
[0x0000002a9e900000, 0x0000002c013b9000, 0x0000002c995d0000)
concurrent-mark-sweep perm gen total 26096K, used 15598K
[0x0000002c995d0000, 0x0000002c9af4c000, 0x0000002c9e9d0000)
....(the rest omitted)
--------------- S Y S T E M ---------------
OS:CentOS release 4.5 (Final)
uname:Linux 2.6.9-55.0.12.EL.nhnsmp #1 SMP Tue Nov 6 18:38:43 KST 2007
x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 16383, NOFILE 32768, AS infinity
load average:1.74 1.51 1.44
CPU:total 4 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping
11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 16407020k(51816k free), swap 4192956k(4016836k
free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05) for linux-amd64,
built on Sep 24 2007 23:09:21 by "java_re" with gcc 3.2.2 (SuSE Linux)
I set java option -Xmx8G and the others were unchanged from Cassandra 0.6.3
settings.
I will try upgrading JVM first.
Thanks for advice. :)
2010/8/3 Jonathan Ellis <jb...@gmail.com>
> If you have a crash log you should post at least the header rather
> than playing 20 questions with us. But if it's not OOM then it's
> likely to be a bug in the JVM, so upgrading is probably your best
> option.
>
> On Tue, Aug 3, 2010 at 3:49 AM, Ilun Ahn <pr...@gmail.com> wrote:
> > No, I don't think the direct cause is out of heap space. It didn't left
> any
> > heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError.
> > My system.log for the last minute is as follows(many GC occurs):
> > INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254
> > HintedHandOffManager.java (line 153) Started hinted handoff for endPoint
> > /10.25.32.36
> > INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255
> > HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to
> > endpoint /10.25.32.36
> > INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 110)
> GC
> > for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 110)
> GC
> > for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 110)
> GC
> > for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 110)
> GC
> > for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 110)
> GC
> > for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 110)
> GC
> > for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458096
> > used; max is 8719630336
> > INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 110)
> GC
> > for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 110)
> GC
> > for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 3550138024
> > used; max is 8719630336
> > INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 110)
> GC
> > for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 110)
> GC
> > for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 110)
> GC
> > for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 110)
> GC
> > for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621048
> > used; max is 8719630336
> > INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 110)
> GC
> > for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 110)
> GC
> > for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 110)
> GC
> > for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is
> > 8719630336
> > INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 110)
> GC
> > for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is
> > 8719630336
> >
> >
> >
> >
> >
> > and the JRE crash log show that current thread is COMPACTION-POOL:1.
> >
> >
> >
> > ps. Sorry for unnecessary message. my mistake.
> >
> >
> >
> >
> > 2010/8/3 Ilun Ahn <pr...@gmail.com>
> >>
> >>
> >> 2010/8/2 Peter Schuller <pe...@infidyne.com>
> >>>
> >>> > First, Cassandra suddenly dies during compaction. Java core dump says
> >>> > that
> >>> > the last thread run was "COMPACTION-POOL:1".
> >>> > I suspect that my business logic could lead size of columns in a
> column
> >>> > family per a row to be greater than two gigabytes. (but i couldn't
> >>> > confirm
> >>> > it yet)
> >>>
> >>> Are you running out of memory (java heap)? If you're running cassandra
> >>> with default options, it will be running with
> >>> -XX:+HeapDumpOnOutOfMemoryError
> >>>
> >>> Have you checked the cassandra system.log for garbage collection
> >>> messages? What is in the last minute or two of logs?
> >>>
> >>> --
> >>> / Peter Schuller
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: Two questions : Server crash during compaction and
UnavailableException
Posted by Jonathan Ellis <jb...@gmail.com>.
If you have a crash log you should post at least the header rather
than playing 20 questions with us. But if it's not OOM then it's
likely to be a bug in the JVM, so upgrading is probably your best
option.
On Tue, Aug 3, 2010 at 3:49 AM, Ilun Ahn <pr...@gmail.com> wrote:
> No, I don't think the direct cause is out of heap space. It didn't left any
> heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError.
> My system.log for the last minute is as follows(many GC occurs):
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254
> HintedHandOffManager.java (line 153) Started hinted handoff for endPoint
> /10.25.32.36
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255
> HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to
> endpoint /10.25.32.36
> INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 110) GC
> for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 110) GC
> for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 110) GC
> for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 110) GC
> for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 110) GC
> for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458096
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 110) GC
> for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 110) GC
> for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 3550138024
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 110) GC
> for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 110) GC
> for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621048
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 110) GC
> for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 110) GC
> for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 110) GC
> for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 110) GC
> for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is
> 8719630336
>
>
>
>
>
> and the JRE crash log show that current thread is COMPACTION-POOL:1.
>
>
>
> ps. Sorry for unnecessary message. my mistake.
>
>
>
>
> 2010/8/3 Ilun Ahn <pr...@gmail.com>
>>
>>
>> 2010/8/2 Peter Schuller <pe...@infidyne.com>
>>>
>>> > First, Cassandra suddenly dies during compaction. Java core dump says
>>> > that
>>> > the last thread run was "COMPACTION-POOL:1".
>>> > I suspect that my business logic could lead size of columns in a column
>>> > family per a row to be greater than two gigabytes. (but i couldn't
>>> > confirm
>>> > it yet)
>>>
>>> Are you running out of memory (java heap)? If you're running cassandra
>>> with default options, it will be running with
>>> -XX:+HeapDumpOnOutOfMemoryError
>>>
>>> Have you checked the cassandra system.log for garbage collection
>>> messages? What is in the last minute or two of logs?
>>>
>>> --
>>> / Peter Schuller
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Two questions : Server crash during compaction and
UnavailableException
Posted by Ilun Ahn <pr...@gmail.com>.
No, I don't think the direct cause is out of heap space. It didn't left any
heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError.
My system.log for the last minute is as follows(many GC occurs):
INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254
HintedHandOffManager.java (line 153) Started hinted handoff for endPoint /
10.25.32.36
INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255
HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to
endpoint /10.25.32.36
INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 110) GC
for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 110) GC
for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 110) GC
for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 110) GC
for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 110) GC
for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 110) GC
for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458096
used; max is 8719630336
INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 110) GC
for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 110) GC
for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 3550138024
used; max is 8719630336
INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 110) GC
for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 110) GC
for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 110) GC
for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 110) GC
for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621048
used; max is 8719630336
INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 110) GC
for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 110) GC
for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 110) GC
for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is
8719630336
INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 110) GC
for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is
8719630336
and the JRE crash log show that current thread is COMPACTION-POOL:1.
ps. Sorry for unnecessary message. my mistake.
2010/8/3 Ilun Ahn <pr...@gmail.com>
>
>
> 2010/8/2 Peter Schuller <pe...@infidyne.com>
>
> > First, Cassandra suddenly dies during compaction. Java core dump says
>> that
>> > the last thread run was "COMPACTION-POOL:1".
>> > I suspect that my business logic could lead size of columns in a column
>> > family per a row to be greater than two gigabytes. (but i couldn't
>> confirm
>> > it yet)
>>
>> Are you running out of memory (java heap)? If you're running cassandra
>> with default options, it will be running with
>> -XX:+HeapDumpOnOutOfMemoryError
>>
>> Have you checked the cassandra system.log for garbage collection
>> messages? What is in the last minute or two of logs?
>>
>> --
>> / Peter Schuller
>>
>
>
Re: Two questions : Server crash during compaction and
UnavailableException
Posted by Ilun Ahn <pr...@gmail.com>.
2010/8/2 Peter Schuller <pe...@infidyne.com>
> > First, Cassandra suddenly dies during compaction. Java core dump says
> that
> > the last thread run was "COMPACTION-POOL:1".
> > I suspect that my business logic could lead size of columns in a column
> > family per a row to be greater than two gigabytes. (but i couldn't
> confirm
> > it yet)
>
> Are you running out of memory (java heap)? If you're running cassandra
> with default options, it will be running with
> -XX:+HeapDumpOnOutOfMemoryError
>
> Have you checked the cassandra system.log for garbage collection
> messages? What is in the last minute or two of logs?
>
> --
> / Peter Schuller
>
Re: Two questions : Server crash during compaction and
UnavailableException
Posted by Peter Schuller <pe...@infidyne.com>.
> First, Cassandra suddenly dies during compaction. Java core dump says that
> the last thread run was "COMPACTION-POOL:1".
> I suspect that my business logic could lead size of columns in a column
> family per a row to be greater than two gigabytes. (but i couldn't confirm
> it yet)
Are you running out of memory (java heap)? If you're running cassandra
with default options, it will be running with
-XX:+HeapDumpOnOutOfMemoryError
Have you checked the cassandra system.log for garbage collection
messages? What is in the last minute or two of logs?
--
/ Peter Schuller