You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Mikhail Stepura (JIRA)" <ji...@apache.org> on 2015/10/17 06:51:05 UTC

[jira] [Comment Edited] (CASSANDRA-10449) OOM on bootstrap after long GC pause

    [ https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961726#comment-14961726 ] 

Mikhail Stepura edited comment on CASSANDRA-10449 at 10/17/15 4:50 AM:
-----------------------------------------------------------------------

So, there are 2 ColumnFamilyStores with 6G  of memtables each (5797 'live' memtables per intsance) in the heap, but for some reason they are not being flushed. All 32 MemtableFlushWriters (and 1 post-flush) are waiting on {{writeBarrier}} s


was (Author: mishail):
So, there are 2  6G memtables in the heap, but for some reason they are not being flushed. All 32 MemtableFlushWriters (and 1 post-flush) are waiting on {{writeBarrier}} s

> OOM on bootstrap after long GC pause
> ------------------------------------
>
>                 Key: CASSANDRA-10449
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 14.04, AWS
>            Reporter: Robbie Strickland
>              Labels: gc
>             Fix For: 2.1.x
>
>         Attachments: GCpath.txt, heap_dump.png, system.log.10-05, thread_dump.log, threads.txt
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to provision additional nodes, but bootstrapping OOMs every time after about 10 hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 CassandraDaemon.java:223 - Exception in thread Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)