[jira] [Commented] (CASSANDRA-13354) LCS estimated compaction tasks does not take number of files into account

["Jan Karlsson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939107#comment-15939107 ] 

Jan Karlsson commented on CASSANDRA-13354:
------------------------------------------

I did some tests simulating traffic on a 4 node cluster. 2 of the nodes were running with my patch while the other two ran without it.
Steps to reproduce:
Traffic on
Turn one of the nodes off
Wait 7 minutes
Truncate hints on all other nodes
Turn node on
Run repair on the node

As you can see the unpatched version kept increasing as non-repaired data from ongoing traffic was prioritized. If I had more discrepancies in my data set, this would just increase to the configured FD limit or until you die from heap pressure.

Repair is completed at 8:11pm but those small repaired files are not compacted as it picks unrepaired new sstables over the small repaired sstables. However, it did show a downwards trend as compaction was slightly faster than insertion and would probably eventually end with the repaired files compacted.

During the unpatched test, it only showed 2 pending compactions with 22k~ file descriptors open/10k~ sstables. At 8:33pm I disabled the traffic completely to hurry this along.
SSTables in each level: [10347/4, 5, 0, 0, 0, 0, 0, 0, 0]

> LCS estimated compaction tasks does not take number of files into account
> -------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13354
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13354
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>         Environment: Cassandra 2.2.9
>            Reporter: Jan Karlsson
>            Assignee: Jan Karlsson
>         Attachments: 13354-trunk.txt, patchedTest.png, unpatchedTest.png
>
>
> In LCS, the way we estimate number of compaction tasks remaining for L0 is by taking the size of a SSTable and multiply it by four. This would give 4*160mb with default settings. This calculation is used to determine whether repaired or repaired data is being compacted.
> Now this works well until you take repair into account. Repair streams over many many sstables which could be smaller than the configured SSTable size depending on your use case. In our case we are talking about many thousands of tiny SSTables. As number of files increases one can run into any number of problems, including GC issues, too many open files or plain increase in read latency.
> With the current algorithm we will choose repaired or unrepaired depending on whichever side has more data in it. Even if the repaired files outnumber the unrepaired files by a large margin.
> Similarily, our algorithm that selects compaction candidates takes up to 32 SSTables at a time in L0, however our estimated task calculation does not take this number into account. These two mechanisms should be aligned with each other.
> I propose that we take the number of files in L0 into account when estimating remaining tasks. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)