You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "GEORGE LI (JIRA)" <ji...@apache.org> on 2019/06/20 20:39:00 UTC
[jira] [Commented] (KAFKA-8527) add dynamic maintenance broker
config
[ https://issues.apache.org/jira/browse/KAFKA-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868914#comment-16868914 ]
GEORGE LI commented on KAFKA-8527:
----------------------------------
We had this implemented in a separate admin service outside of kafka. One of its functions is for topic related changes. e.g. Topic Creation, Topic partition expansion. There is a ZK path/node which indicate which broker_id is under "maintenance" and thus when creating topic/expanding topic partition, it will exclude this live broker_id from getting any new replica assignments.
> add dynamic maintenance broker config
> -------------------------------------
>
> Key: KAFKA-8527
> URL: https://issues.apache.org/jira/browse/KAFKA-8527
> Project: Kafka
> Issue Type: Improvement
> Reporter: xiongqi wu
> Assignee: xiongqi wu
> Priority: Major
>
> Before we remove a broker for maintenance, we want to remove all partitions out of the broker first to avoid introducing new Under Replicated Partitions (URPs) . That is because shutting down (or killing) a broker that still hosts live partitions will lead to temporarily reduced replicas of those partitions. Moving partitions out of a broker can be done via partition reassignment. However, during the partition reassignment process, new topics can be created by Kafka and thereby new partitions can be added to the broker that is pending for removal. As a result, the removal process will need to recursively moving new topic partitions out of the maintenance broker. In a production environment in which topic creation is frequent and URP causing by broker removal cannot be tolerated, the removal process can take multiple iterations to complete the partition reassignment. We want to provide a mechanism to mask a broker as maintenance broker (Via Cluster Level Dynamic configuration). One action Kafka can take for the maintenance broker is not to assign new topic partitions to it, and thereby facilitate the broker removal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)