[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

["K Zakee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371402#comment-15371402 ] 

K Zakee commented on KAFKA-1464:
--------------------------------

I agree with Ralph. 
Lets say, we have a high produce rate and a system failure (as long as the kafka retention period itself), there is a lot of data to catchup and as fast as it could. Throttling catching up of out-of-sync replicas in this case may become a "chase-your-own-tail thing" and these may never be able to catchup with their leader or take days depending on produce-rate and throttle limit. Suppressing new replicas taking the leadership until the time they have all caught up sounds a better idea.

> Add a throttling option to the Kafka replication tool
> -----------------------------------------------------
>
>                 Key: KAFKA-1464
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1464
>             Project: Kafka
>          Issue Type: New Feature
>          Components: replication
>    Affects Versions: 0.8.0
>            Reporter: mjuarez
>            Assignee: Ben Stopford
>            Priority: Minor
>              Labels: replication, replication-tools
>             Fix For: 0.10.1.0
>
>
> When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible.  This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster.
> An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)