[jira] [Commented] (HBASE-8329) Limit compaction speed

["zhangduo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251252#comment-14251252 ] 

zhangduo commented on HBASE-8329:
---------------------------------

I met the problem recently. Sometimes "Compaction progress" report the rate is about 40MB/sec，so if we have two compaction in progess and other operation together then we can easily exceed a 1000Mbit/sec network card's max bandwith, and the regionserver will be unresponsive.
I guess the reason is that we have some large rows(several hundreds KBs to several MBs), so the compaction speed can easily burst high. I can do a test to examine how does row size effect the compaction speed. And our cluster is small(7 regionserver. A machine can have 30TB disk capacity today, and IT department calculate the number of regionservers by disk capacity, not by memory and network:( ) so the total bandwith is small.
I apply the patch to 0.98.9RC1 to limit the compaction speed. It works for now but I don't think it should be the final solution. At least to we need to calculate the limitation automatically I think.

> Limit compaction speed
> ----------------------
>
>                 Key: HBASE-8329
>                 URL: https://issues.apache.org/jira/browse/HBASE-8329
>             Project: HBase
>          Issue Type: Improvement
>          Components: Compaction
>            Reporter: binlijin
>            Assignee: Sergey Shelukhin
>             Fix For: 2.0.0, 1.1.0
>
>         Attachments: HBASE-8329-10.patch, HBASE-8329-11.patch, HBASE-8329-12.patch, HBASE-8329-2-trunk.patch, HBASE-8329-3-trunk.patch, HBASE-8329-4-trunk.patch, HBASE-8329-5-trunk.patch, HBASE-8329-6-trunk.patch, HBASE-8329-7-trunk.patch, HBASE-8329-8-trunk.patch, HBASE-8329-9-trunk.patch, HBASE-8329-trunk.patch
>
>
> There is no speed or resource limit for compaction，I think we should add this feature especially when request burst.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)