You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Luke Chen (Jira)" <ji...@apache.org> on 2025/11/13 06:47:00 UTC
[jira] [Updated] (KAFKA-19850) KRaft voter auto join will add a removed voter immediately
[ https://issues.apache.org/jira/browse/KAFKA-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Chen updated KAFKA-19850:
------------------------------
Description:
In v4.2.0, we are able to auto join a controller with the configuration `controller.quorum.auto.join.enable=true` set ([KIP-853|https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes#KIP853:KRaftControllerMembershipChanges-Controllerautojoining](KAFKA-19078)). This is a good improvement for controller addition, but it has a UX issue, which is that when a controller is removed via removeVoterRequest, it will be added immediately due to `controller.quorum.auto.join.enable=true`. In the KIP, we also mention you have to stop the controller before removing the controller:
{noformat}
controller.quorum.auto.join.enable:
Controls whether a KRaft controller should automatically join the cluster
metadata partition for its cluster id. If the configuration is set to
true the controller must be stopped before removing the controller with kafka-metadata-quorum remove-controller.{noformat}
This "shutdown the to-be-removed controller first" operation might break the quorum in the worst case. For example, 3 controller nodes quorum (C1, C2, C3), C1 is the leader, C3 is already caught up with C1, C2 is still catching up with the leader. When users want to remove C3, following the guide, users shutdown the C3 first. But at this point of time, the quorum is broken and the kafka cluster is basically unavailable.
Furthermore, this is not a user friendly behavior. And it will cause many confusion to users and thought there is something wrong in the controller removal. Furthermore, in the kubernetes environment which is controlled by the operator, it is not the cloud native way to shutdown a node, do some operation, then start it up.
So, I propose we can improve it by "the removed controller will not be auto joined before this controller restarted". That is:
1. Once the controller is removed from voters set, it won't be auto joined even if `controller.quorum.auto.join.enable=true`
2. The controller can be manually join the voters in this state
3. The controller node will be auto join the voters set after node restarted.
So basically, the semantics is not changed, it just add some unexpected remove/add loop. Thoughts?
was:
In v4.2.0, we are able to auto join a controller with the configuration `controller.quorum.auto.join.enable=true` set ([KIP-853|https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes#KIP853:KRaftControllerMembershipChanges-Controllerautojoining](KAFKA-19078)). This is a good improvement for controller addition, but it has a UX issue, which is that when a controller is removed via removeVoterRequest, it will be added immediately due to `controller.quorum.auto.join.enable=true`. In the KIP, we also mention you have to stop the controller before removing the controller:
{noformat}
controller.quorum.auto.join.enable:
Controls whether a KRaft controller should automatically join the cluster
metadata partition for its cluster id. If the configuration is set to
true the controller must be stopped before removing the controller with kafka-metadata-quorum remove-controller.{noformat}
This is not a user friendly behavior in my opinion. And it will cause many confusion to users and thought there is something wrong in the controller removal. Furthermore, in the kubernetes environment which is controlled by the operator, it is not the cloud native way to shutdown a node, do some operation, then start it up.
So, I propose we can improve it by "the removed controller will not be auto joined before this controller restarted". That is:
1. Once the controller is removed from voters set, it won't be auto joined even if `controller.quorum.auto.join.enable=true`
2. The controller can be manually join the voters in this state
3. The controller node will be auto join the voters set after node restarted.
So basically, the semantics is not changed, it just add some unexpected remove/add loop. Thoughts?
> KRaft voter auto join will add a removed voter immediately
> ----------------------------------------------------------
>
> Key: KAFKA-19850
> URL: https://issues.apache.org/jira/browse/KAFKA-19850
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 4.2.0
> Reporter: Luke Chen
> Priority: Major
>
> In v4.2.0, we are able to auto join a controller with the configuration `controller.quorum.auto.join.enable=true` set ([KIP-853|https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes#KIP853:KRaftControllerMembershipChanges-Controllerautojoining](KAFKA-19078)). This is a good improvement for controller addition, but it has a UX issue, which is that when a controller is removed via removeVoterRequest, it will be added immediately due to `controller.quorum.auto.join.enable=true`. In the KIP, we also mention you have to stop the controller before removing the controller:
>
> {noformat}
> controller.quorum.auto.join.enable:
> Controls whether a KRaft controller should automatically join the cluster
> metadata partition for its cluster id. If the configuration is set to
> true the controller must be stopped before removing the controller with kafka-metadata-quorum remove-controller.{noformat}
>
> This "shutdown the to-be-removed controller first" operation might break the quorum in the worst case. For example, 3 controller nodes quorum (C1, C2, C3), C1 is the leader, C3 is already caught up with C1, C2 is still catching up with the leader. When users want to remove C3, following the guide, users shutdown the C3 first. But at this point of time, the quorum is broken and the kafka cluster is basically unavailable.
> Furthermore, this is not a user friendly behavior. And it will cause many confusion to users and thought there is something wrong in the controller removal. Furthermore, in the kubernetes environment which is controlled by the operator, it is not the cloud native way to shutdown a node, do some operation, then start it up.
>
> So, I propose we can improve it by "the removed controller will not be auto joined before this controller restarted". That is:
> 1. Once the controller is removed from voters set, it won't be auto joined even if `controller.quorum.auto.join.enable=true`
> 2. The controller can be manually join the voters in this state
> 3. The controller node will be auto join the voters set after node restarted.
>
> So basically, the semantics is not changed, it just add some unexpected remove/add loop. Thoughts?
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)