[jira] [Commented] (CASSANDRA-5519) Reduce index summary memory use for cold sstables

["Tyler Hobbs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820653#comment-13820653 ] 

Tyler Hobbs commented on CASSANDRA-5519:
----------------------------------------

I need to put this through more thorough testing and benchmarking, but I think it's at a good point for a preliminary review: https://github.com/thobbs/cassandra/compare/CASSANDRA-5519

A few comments/questions:
* I went with best-effort for the memory pool (if all summaries don't fit in the allotted space even at the minimum sampling level, there's nothing we can do about it).  The amount of memory used may also temporarily exceed the limit while building new summaries.
* There are two new cassandra.yaml options: one for controlling the memory pool size and one for regulating how frequently summaries are resized.  These can also be set through JMX. We could conceivably also make the down/upsample thresholds and the minimum sampling level configurable.  All of these default values are just guesses.
* I went with a reference counting strategy for free'ing the IndexSummary's Memory.  This makes the API a bit unpleasant (mostly in SSTR), but it should have low overhead.  A ReadWriteLock might also work well instead of this with a cleaner API; let me know if I should benchmark the two for comparison.
* I'm triggering the IndexSummaryManager singleton's initialization in DatabaseDescriptor; this feels wrong, so I'm open to suggestions.

> Reduce index summary memory use for cold sstables
> -------------------------------------------------
>
>                 Key: CASSANDRA-5519
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5519
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Tyler Hobbs
>            Priority: Minor
>             Fix For: 2.1
>
>         Attachments: downsample.py
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)