[jira] [Comment Edited] (CASSANDRA-14239) OutOfMemoryError when bootstrapping with less than 100GB RAM

["Sergey Kirillov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392951#comment-16392951 ] 

Sergey Kirillov edited comment on CASSANDRA-14239 at 3/9/18 3:08 PM:
---------------------------------------------------------------------

It fails during bootstrap, it fails if I skip bootstrap and doing repair.

memtable size set to 
{code:yaml}
memtable_heap_space_in_mb: 1048
memtable_offheap_space_in_mb: 1048
{code}
and 
{code:yaml}
memtable_flush_writers: 16
{code}

When I analyze heap dump I see that 95% of memory is used by Memtable instances. There are 24k instances of Memtable class and their retained heap is 26G, but they have no GC root. 

So this means that they must be garbage collected, I don't understand why I'm getting OOM instead.


was (Author: rushman):
It fails during bootstrap, it fails if I skip bootstrap and doing repair.

memtable size set to 
{code:yaml}
memtable_heap_space_in_mb: 1048
memtable_offheap_space_in_mb: 1048
{code}
and 
{code:yaml}
memtable_flush_writers: 16
{code}

When I analyze heap dump I see that 95% of memory is used by Memtable instances. There are 24k instances of Memtable class and their retained heap is 26G.

> OutOfMemoryError when bootstrapping with less than 100GB RAM
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-14239
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14239
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Details of the bootstrapping Node
>  * ProLiant BL460c G7
>  * 56GB RAM
>  * 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and saved_caches)
>  * CentOS 7.4 on SD-Card
>  * /tmp and /var/log on tmpfs
>  * Oracle JDK 1.8.0_151
>  * Cassandra 3.11.1
> Cluster
>  * 10 existing Nodes (Up and Normal)
>            Reporter: Jürgen Albersdorfer
>            Priority: Major
>         Attachments: Objects-by-class.csv, Objects-with-biggest-retained-size.csv, cassandra-env.sh, cassandra.yaml, jvm.options, jvm_opts.txt, stack-traces.txt
>
>
> Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed up any more.
> I know that JVM collects on Old Gen only when really needed. I can see collections, but there is always a remainder which seems to grow forever without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra RAM I have given it for bootstrapping without any further effect.
> It feels like Cassandra will not forget about every single byte streamed over the Network over time during bootstrapping, - which would be a memory leak and a major problem, too.
> I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated for org.apache.cassandra.db.Memtable (22 GB) org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11 GB)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org