You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "James Cheng (JIRA)" <ji...@apache.org> on 2016/07/11 20:48:11 UTC
[jira] [Comment Edited] (KAFKA-1464) Add a throttling option to the
Kafka replication tool
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371578#comment-15371578 ]
James Cheng edited comment on KAFKA-1464 at 7/11/16 8:47 PM:
-------------------------------------------------------------
[~r.weires], [~kzakee], you might be able to control this a little by setting auto.leader.rebalance.enable=false. If you it to false, then the broker would come up but would not assume leadership for any partitions at all, unless manually told to. You would then have to use the kafka-preferred-replica-election.sh tool [https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool], to allow it to assume leadership.
This would mean that you wouldn't have the problem you described. But the downside is that you are now in charge of handling rebalancing on your own.
The auto.leader.rebalance.enable flag is not changeable during runtime, tho. I think it is only read at startup time.
was (Author: wushujames):
[~r.weires], you might be able to control this a little by setting auto.leader.rebalance.enable=false. If you it to false, then the broker would come up but would not assume leadership for any partitions at all, unless manually told to. You would then have to use the kafka-preferred-replica-election.sh tool [https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool], to allow it to assume leadership.
This would mean that you wouldn't have the problem you described. But the downside is that you are now in charge of handling rebalancing on your own.
The auto.leader.rebalance.enable flag is not changeable during runtime, tho. I think it is only read at startup time.
> Add a throttling option to the Kafka replication tool
> -----------------------------------------------------
>
> Key: KAFKA-1464
> URL: https://issues.apache.org/jira/browse/KAFKA-1464
> Project: Kafka
> Issue Type: New Feature
> Components: replication
> Affects Versions: 0.8.0
> Reporter: mjuarez
> Assignee: Ben Stopford
> Priority: Minor
> Labels: replication, replication-tools
> Fix For: 0.10.1.0
>
>
> When performing replication on new nodes of a Kafka cluster, the replication process will use all available resources to replicate as fast as possible. This causes performance issues (mostly disk IO and sometimes network bandwidth) when doing this in a production environment, in which you're trying to serve downstream applications, at the same time you're performing maintenance on the Kafka cluster.
> An option to throttle the replication to a specific rate (in either MB/s or activities/second) would help production systems to better handle maintenance tasks while still serving downstream applications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)