[jira] [Commented] (CASSANDRA-2191) Multithread across compaction buckets

["Aaron Morton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015150#comment-13015150 ] 

Aaron Morton commented on CASSANDRA-2191:
-----------------------------------------

4) Sounds reasonable if throttling is on.
 
6) I'm not familiar with the bloom filter optimization you mentioned. However it seems that more than anything else the major flag in doCompaction() indicates if the compaction is running on all sstables, regardless of how the process was triggered. i.e. the first ever minor compaction would also be marked as major by this logic. PrecompactedRow and LazilyCompactedRow will purge rows if the major flag is set or the key is only present in the sstables under compaction. I'm not sure why the extra check is there for minor compactions, but it looks like losing the fact the a major/manual compaction was started could change the purge behaviour. 

I'm also trying to understand if the isKeyInRemainingSSTables() in the AbstractCompactedRow sub classes could be affected by multithreading. e.g. CF with two buckets, high min compaction threshold so longer compaction, two concurrent minor compactions one in each bucket, row A in both buckets, if either thread processes row A before the other finishes it would stop that thread purging the row, is there a race condition that stops both threads purging the row?

9) We do not use the value in the compactions map, could we set it to the current system time when beginCompaction() is called and use that to the sort the list ? was not a biggie 



> Multithread across compaction buckets
> -------------------------------------
>
>                 Key: CASSANDRA-2191
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2191
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Stu Hood
>            Assignee: Stu Hood
>            Priority: Critical
>              Labels: compaction
>             Fix For: 0.8
>
>         Attachments: 0001-Add-a-compacting-set-to-DataTracker.txt, 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt, 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt, 0004-Try-harder-to-close-scanners-in-compaction-close.txt
>
>
> This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and reasoning are different enough to open a separate issue.
> The problem with compactions currently is that they compact the set of sstables that existed the moment the compaction started. This means that for longer running compactions (even when running as fast as possible on the hardware), a very large number of new sstables might be created in the meantime. We have observed this proliferation of sstables killing performance during major/high-bucketed compactions.
> One approach would be to pause compactions in upper buckets (containing larger files) when compactions in lower buckets become possible. While this would likely solve the problem with read performance, it does not actually help us perform compaction any faster, which is a reasonable requirement for other situations.
> Instead, we need to be able to perform any compactions that are currently required in parallel, independent of what bucket they might be in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira