[jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions

["Michael Kjellman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15434219#comment-15434219 ] 

Michael Kjellman commented on CASSANDRA-9754:
---------------------------------------------

So, I'm mostly done with a trunk version of the patch, however, I'm currently focusing on finishing and polishing the 2.1 based version. Although the abstraction of the index is almost a total rewrite between 2.1 and trunk the tree itself and the Birch implementation should remain the same so this certainly isn't wasted time for anyone. :) I've cleaned up the implementation a bunch, taken care of a bunch of todos and low hanging fruit, added more documentation, and pushed it to Github to make it a bit easier to make sure the changes apply cleanly.

https://github.com/mkjellman/cassandra/commit/e5389378b19eb03de7dd4d50d6df110c68057985

The following 4 unit tests (out of 1184) are still failing (so close!):
 * org.apache.cassandra.cql3.KeyCacheCqlTest (2 of 2). Need to talk to [~aweisberg] to understand exactly what these unit tests are testing.
 * org.apache.cassandra.db.ColumnFamilyStoreTest (2 of 38, both related to secondary indexes)

Tomorrow, I hope to push a patch addressing the feedback from [~barnie] (see above comment) above along with any changes that come out of the code review currently underway by [~jasobrown] and [~kohlisankalp]. I also need/want to do some work on feeling more comfortable on the upgrade/backwards compatibility story and make sure there is a good unit test story around that.

[~jjirsa] if you get a chance to take a look please let me know if you have any initial feedback that would be awesome!

> Make index info heap friendly for large CQL partitions
> ------------------------------------------------------
>
>                 Key: CASSANDRA-9754
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9754
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: sankalp kohli
>            Assignee: Michael Kjellman
>            Priority: Minor
>             Fix For: 4.x
>
>         Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff
>
>
>  Looking at a heap dump of 2.0 cluster, I found that majority of the objects are IndexInfo and its ByteBuffers. This is specially bad in endpoints with large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for GC. Can this be improved by not creating so many objects?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)