You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Zhiyan Shao <zh...@gmail.com> on 2016/05/21 05:02:35 UTC
OOM under high write throughputs on 2.2.5
Hi, we see the following OOM crash while doing heavy write loading testing.
Has anybody seen this kind of crash? We are using G1GC with 32GB heap size
out of 128GB system memory. Eclipse Memory Analyzer shows the following:
One instance of *"org.apache.cassandra.db.ColumnFamilyStore"* loaded
by *"sun.misc.Launcher$AppClassLoader
@ 0x8d800898"* occupies *697,236,904 (51.91%)* bytes. The memory is
accumulated in one instance of
*"java.util.concurrent.ConcurrentSkipListMap$HeadIndex"* loaded by *"<system
class loader>"*.
*Keywords*
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
sun.misc.Launcher$AppClassLoader @ 0x8d800898
org.apache.cassandra.db.ColumnFamilyStore
Cassandra log:
ERROR 00:23:24 JVM state determined to be unstable. Exiting forcefully due
to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_74]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_74]
at
org.apache.cassandra.utils.memory.SlabAllocator.getRegion(SlabAllocator.java:
137) ~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:
97) ~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:
57) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.memory.ContextAllocator.clone
(ContextAllocator.java:47) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.utils.memory.MemtableBufferAllocator.clone
(MemtableBufferAllocator.java:61) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Memtable.put(Memtable.java:212)
~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:
1249) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:406)
~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:366)
~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:
50) ~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-2.2.5.jar:2.2.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_74]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:
164) ~[apache-cassandra-2.2.5.jar:2.2.5]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:
136) [apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.2.5.jar:2.2.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Thanks,
Zhiyan
Re: OOM under high write throughputs on 2.2.5
Posted by Bryan Cheng <br...@blockcypher.com>.
Hi Zhiyan,
Silly question but are you sure your heap settings are actually being
applied? "697,236,904 (51.91%)" would represent a sub-2GB heap. What's the
real memory usage for Java when this crash happens?
Other thing to look into might be memtable_heap_space_in_mb, as it looks
like you're using onheap memtables. This will be 1/4 of your heap by
default. Assuming your heap settings are actually being applied, if you run
through this space you may not have enough flushing resources.
memtable_flush_Writers defaults to a somewhat low number which may not be
enough for this use case.
On Fri, May 20, 2016 at 10:02 PM, Zhiyan Shao <zh...@gmail.com> wrote:
> Hi, we see the following OOM crash while doing heavy write loading
> testing. Has anybody seen this kind of crash? We are using G1GC with 32GB
> heap size out of 128GB system memory. Eclipse Memory Analyzer shows the
> following:
>
> One instance of *"org.apache.cassandra.db.ColumnFamilyStore"* loaded by *"sun.misc.Launcher$AppClassLoader
> @ 0x8d800898"* occupies *697,236,904 (51.91%)* bytes. The memory is
> accumulated in one instance of
> *"java.util.concurrent.ConcurrentSkipListMap$HeadIndex"* loaded by *"<system
> class loader>"*.
>
> *Keywords*
>
> java.util.concurrent.ConcurrentSkipListMap$HeadIndex
>
> sun.misc.Launcher$AppClassLoader @ 0x8d800898
>
> org.apache.cassandra.db.ColumnFamilyStore
>
> Cassandra log:
>
>
> ERROR 00:23:24 JVM state determined to be unstable. Exiting forcefully
> due to:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_74]
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_74]
> at
> org.apache.cassandra.utils.memory.SlabAllocator.getRegion(SlabAllocator.java:
> 137) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:
> 97) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:
> 57) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.memory.ContextAllocator.clone
> (ContextAllocator.java:47) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.utils.memory.MemtableBufferAllocator.clone
> (MemtableBufferAllocator.java:61) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Memtable.put(Memtable.java:212)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:
> 1249) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:406)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:366)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:
> 50) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:
> 67) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_74]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:
> 164) ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:
> 136) [apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.2.5.jar:2.2.5]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> Thanks,
> Zhiyan
>