[jira] [Commented] (CASSANDRA-12366) Fix compaction throttle

["T Jake Luciani (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404623#comment-15404623 ] 

T Jake Luciani commented on CASSANDRA-12366:
--------------------------------------------

[branch|https://github.com/tjake/cassandra/tree/fix-compaction-ratelimit]
[testall|http://cassci.datastax.com/view/Dev/view/tjake/job/tjake-fix-compaction-ratelimit-testall/]
[dtest|http://cassci.datastax.com/view/Dev/view/tjake/job/tjake-fix-compaction-ratelimit-dtest/]

To show the fix:
  I set a compaction limit of 8mb/sec

Before fix:
{code}
 459.971MiB to 458.545MiB (~99% of original) in 99,556ms.  Read Throughput = 4.620MiB/s, Write Throughput = 4.606MiB/s, Row Throughput = ~298,272/s.  30,083 total partitions merged to 30,078.  Partition merge counts were {1:30073, 2:5, }
{code}

Branch fix:
{code}
 459.971MiB to 458.545MiB (~99% of original) in 57,124ms.  Read Throughput = 8.052MiB/s, Write Throughput = 8.027MiB/s, Row Throughput = ~514,263/s.  30,083 total partitions merged to 30,078.  Partition merge counts were {1:30073, 2:5, }
{code}


The output of compactionstats and CompactionInfo is still showing the uncompressed data size but I see no simple way to fetching the actual on disk pointer after CASSANDRA-5863


> Fix compaction throttle
> -----------------------
>
>                 Key: CASSANDRA-12366
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12366
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>             Fix For: 3.x
>
>
> Compaction throttling is broken in the following ways:
>   * It throttles bytes read after being decompressed
>   * Compaction creates multiple scanners which share the rate limiter causing too much throttling
>   * It bears no resemblance to the reported compaction time remaining calculation (Bytes of source sstables processed since start of compaction)
> To fix this we need to simplify the throttling to be only at the CompactionIterator level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)