[jira] [Commented] (CASSANDRA-12201) Burst Hour Compaction Strategy

["Pedro Gordo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049671#comment-16049671 ] 

Pedro Gordo commented on CASSANDRA-12201:
-----------------------------------------

I've squashed several commits, and added everything to a fork from the proper cassandra repo. You can find it here: https://github.com/sedulam/cassandra/tree/12201
I believe now you can easily compare my changes to the code base. Please let me know if this should be done differently.

> Burst Hour Compaction Strategy
> ------------------------------
>
>                 Key: CASSANDRA-12201
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12201
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Compaction
>            Reporter: Pedro Gordo
>         Attachments: BHCS outline.pdf
>
>   Original Estimate: 1,008h
>  Remaining Estimate: 1,008h
>
> This strategy motivation revolves around taking advantage of periods of the day where there's less I/O on the cluster. This time of the day will be called “Burst Hour” (BH), and hence the strategy will be named “Burst Hour Compaction Strategy” (BHCS). 
> The following process would be fired during BH:
> 1. Read all the SSTables and detect which partition keys are present in more than the compaction minimum threshold value.
> 2. Gather all the tables that have keys present in other tables, with a minimum of replicas equal to the minimum compaction threshold. 
> 3. Repeat step 2 until the bucket for gathered SSTables reaches the maximum compaction threshold (32 by default), or until we've searched all the keys.
> 4. The compaction per se will be done through by MaxSSTableSizeWriter. The compacted tables will have a maximum size equal to the configurable value of max_sstable_size (100MB by default). 
> The maximum compaction task (nodetool compact command), does exactly the same operation as the background compaction task, but differing in that it can be triggered outside of the Burst Hour.
> This strategy tries to address three issues of the existing compaction strategies:
> - Due to max_sstable_size_limit, there's no need to reserve disc space for a huge compaction.
> - The number of SSTables that we need to read from to reply to a read query will be consistently maintained at a low level and controllable through the referenced_sstable_limit property.
> - It removes the dependency of a continuous high I/O.
> Possible future improvements:
> - Continuously evaluate how many pending compactions we have and I/O status, and then based on that, we start (or not) the compaction.
> - If during the day, the size for all the SSTables in a family set reaches a certain maximum, then background compaction can occur anyway. This maximum should be elevated due to the high CPU usage of BHCS.
> - Make it possible to set several compaction times intervals, instead of just one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org