[jira] [Updated] (CASSANDRA-16318) Memtable heap size is severely underestimated

["Benjamin Lerer (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-16318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Lerer updated CASSANDRA-16318:
---------------------------------------
     Bug Category: Parent values: Correctness(12982)
       Complexity: Normal
    Discovered By: User Report
    Fix Version/s: 4.0-beta
                   3.11.x
                   3.0.x
         Severity: Normal
           Status: Open  (was: Triage Needed)

> Memtable heap size is severely underestimated
> ---------------------------------------------
>
>                 Key: CASSANDRA-16318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16318
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Memtable
>            Reporter: Branimir Lambov
>            Priority: Normal
>             Fix For: 3.0.x, 3.11.x, 4.0-beta
>
>         Attachments: image-2020-12-09-10-57-21-994.png, image-2020-12-09-11-01-31-273.png
>
>
> We seem to be estimating the size of the on-heap memtable metadata to be around half of what it actually is. For example, during a [read benchmark which writes 1 million single-long rows|https://github.com/blambov/cassandra/blob/memtable-heap/test/microbench/org/apache/cassandra/test/microbench/instance/ReadTestSmallPartitions.java] the memtable reports
> {code}
> 1000000 ops, 58.174MiB serialized bytes, 385.284MiB (19%) on heap, 0.000KiB (0%) off-heap
> {code}
> while a heap dump taken at this point:
>  !image-2020-12-09-10-57-21-994.png! 
> lists an usage of about 666MB altogether.
> Switching to {{offheap_objects}}, the reported numbers are
> {code}
> 1000000 ops, 58.174MiB serialized bytes, 233.650MiB (12%) on heap, 53.406MiB (3%) off-heap
> {code}
> while actual heap usage:
>  !image-2020-12-09-11-01-31-273.png! 
> is about 442MB.
> Looking at the code we definitely are not counting the {{AtomicBTreePartition.Holder}}, {{EncodingStats}}, liveness and deletion info objects associated with each partition, and most probably others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org