[jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions

["Michael Kjellman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880077#comment-15880077 ] 

Michael Kjellman commented on CASSANDRA-9754:
---------------------------------------------

Just wanted to give a quick update:

# I'm really sorry for the delay getting this finished for trunk. I've started a trunk based/post 8099 version 3 times now -- the holidays happened -- more pressing things stole my attention -- big commits like removing Thrift, CFMetadata, etc, etc kept getting committed before I was done, and well -- enough excuses from me...
# A belated thanks for your initial comments Branimir -- I did read them and I'll be addressing them with my trunk rebased changes.
# I'm almost done with the refactoring to move all the current array based index logic into a IndexEntry implementation. I have all unit tests passing (finally) with the exception of KeyCacheCqlTest (which I'm working on right now).
# Assuming I get the post 8099 Indexed Iterator based abstractions/changes correct it should be a matter of just dropping in the Birch package/classes I had for 2.1 and switching the default serializer to use the Birch IndexedEntry implementation.


> Make index info heap friendly for large CQL partitions
> ------------------------------------------------------
>
>                 Key: CASSANDRA-9754
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9754
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: sankalp kohli
>            Assignee: Michael Kjellman
>            Priority: Minor
>             Fix For: 4.x
>
>         Attachments: 0f8e28c220fd5af6c7b5dd2d3dab6936c4aa4b6b.patch, gc_collection_times_with_birch.png, gc_collection_times_without_birch.png, gc_counts_with_birch.png, gc_counts_without_birch.png, perf_cluster_1_with_birch_read_latency_and_counts.png, perf_cluster_1_with_birch_write_latency_and_counts.png, perf_cluster_2_with_birch_read_latency_and_counts.png, perf_cluster_2_with_birch_write_latency_and_counts.png, perf_cluster_3_without_birch_read_latency_and_counts.png, perf_cluster_3_without_birch_write_latency_and_counts.png
>
>
>  Looking at a heap dump of 2.0 cluster, I found that majority of the objects are IndexInfo and its ByteBuffers. This is specially bad in endpoints with large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for GC. Can this be improved by not creating so many objects?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)