You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Paolo Patierno (Jira)" <ji...@apache.org> on 2025/11/05 12:56:00 UTC

[jira] [Created] (KAFKA-19867) Broker only node sending UpdateVoteRequest when it can't really become a voter

Paolo Patierno created KAFKA-19867:
--------------------------------------

             Summary: Broker only node sending UpdateVoteRequest when it can't really become a voter
                 Key: KAFKA-19867
                 URL: https://issues.apache.org/jira/browse/KAFKA-19867
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 4.2.0
            Reporter: Paolo Patierno


I am in the process of implementing support for controllers scaling within the Strimzi project (running Apache Kafka on Kubernetes) by also using the Apache Kafka code in the current "trunk" branch so the future 4.2.0 release because I want to leverage the auto-join feature.

When scaling down controllers, the auto-join related documentation mentions that you should first shutdown the controller and later running the remove-controller (via the kafka-metadata-quorum tool, or programmatically in a Kubernetes operator case by using the RemoveRaftVoter via the Admin Client API), otherwise it's pretty clear the node enters in a loop where you remove it but it rejoins automatically again. 
When managing a Kafka cluster running on bare metal/VMs, this approach works fine even in case the controller scale-down is happening by removing the controller role from a mixed node (shutdown the node, run kafka-metadata-quorum tool to remove-controller, restart the node as broker only). But in a cloud-native environment like Kubernetes, the pod rolling is driven by the platform so there is no way to run a RemoveRaftVoter admin call in between the shutdown and restart. For this reason, the remove-controller is done when the node restarts as broker only.

The issue I am facing is that when such a node restarts as broker only, but it's still in the quorum voter (because the remove-controller isn't happened yet), I get the following exception:
 
{code:java}
2025-10-31 08:01:21 TRACE [kafka-1-raft-io-thread] KafkaRaftClient:2899 - [RaftManager id=1] Sent outbound request: OutboundRequest(correlationId=13, data=UpdateRaftVoterRequestData(clusterId='zsn8QaOzTICYZBhUYQpJBg', currentLeaderEpoch=2, voterId=1, voterDirectoryId=ceZ1jCL9DirrUuCxwsv-jw, listeners=[], kRaftVersionFeature=KRaftVersionFeature(minSupportedVersion=0, maxSupportedVersion=1)), createdTimeMs=1761897681990, destination=my-cluster-broker-0.my-cluster-kafka-brokers.myproject.svc:9090 (id: 0 rack: null isFenced: false))2025-10-31 08:01:21 TRACE [kafka-1-raft-io-thread] KafkaRaftClient:2830 - [RaftManager id=1] Received inbound message InboundResponse(correlationId=13, data=UpdateRaftVoterResponseData(throttleTimeMs=0, errorCode=42, currentLeader=CurrentLeader(leaderId=0, leaderEpoch=2, host='my-cluster-broker-0.my-cluster-kafka-brokers.myproject.svc', port=9090)), source=my-cluster-broker-0.my-cluster-kafka-brokers.myproject.svc:9090 (id: 0 rack: null isFenced: false))2025-10-31 08:01:21 ERROR [kafka-1-raft-io-thread] ProcessTerminatingFaultHandler:46 - Encountered fatal fault: Unexpected error in raft IO threadjava.lang.IllegalStateException: Received unexpected invalid request error	at org.apache.kafka.raft.KafkaRaftClient.maybeHandleCommonResponse(KafkaRaftClient.java:2679)	at org.apache.kafka.raft.KafkaRaftClient.handleUpdateVoterResponse(KafkaRaftClient.java:2569)	at org.apache.kafka.raft.KafkaRaftClient.handleResponse(KafkaRaftClient.java:2737)	at org.apache.kafka.raft.KafkaRaftClient.handleInboundMessage(KafkaRaftClient.java:2836)	at org.apache.kafka.raft.KafkaRaftClient.poll(KafkaRaftClient.java:3680)	at org.apache.kafka.raft.KafkaRaftClientDriver.doWork(KafkaRaftClientDriver.java:64)	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136) {code}

It's happening because the node (which is now broker only) is sending a UpdateRaftVoter request (because it sees itself still in the voters list) even if it's not actually a controller and, of course, it's not able to handle the response which is unexpected because it's a broker only node.

I think, despite the remove-controller was not done yet, the broker-only node should not send such a request even because in any case it's not able to handle the response so it ends in a "broken" code path.

The code where it's happening is within the {{KafkaRaftClient.shouldSendUpdateVoteRequest}} where it's not checking the {{canBecomeVoter}} flag before sending the request (here https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L3299). Such a check is available in the {{shouldSendAddOrRemoveVoterRequest}} method instead (here https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L3355).

I think that adding the check would fix the issue because actually the node is not a controller anymore and it can't really become a voter and that flag is, of course, false avoiding the node to send the UpdateRaftVoter request.

If accepted, I would be willing to open a PR to fix this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)