[jira] [Comment Edited] (KAFKA-20427) Controllers can't form quorum on restart when hostnames changes with dynamic quorum

["José Armando García Sancio (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18073304#comment-18073304 ] 

José Armando García Sancio edited comment on KAFKA-20427 at 4/14/26 2:23 AM:
-----------------------------------------------------------------------------

[~ppatierno] thanks for the additional information. I have a couple of suggestions. At a high-level the controllers need to be able to reach each other at all times. You can do this by:
 # In DNS or /etc/hosts make sure that the old and new names resolve correctly. That means that you need entries for controller-# and controller-#-new
 # Update the advertised.listeners and controller.quorum.bootstrap.servers one node at at time. Note that both controllers and brokers need to have the updated controller.quorum.bootstrap.servers
 # Once all of the nodes have been updated, in the DNS or /etc/hosts remove the previous or old names: controller-#

Let me know if that works for you. If so, I am willing to review a change to the documentation that describe this process.


was (Author: jsancio):
[~ppatierno] thanks for the additional information. I have a couple of suggestions. At a high-level the controllers need to be able to reach each other at all times. You can do this by:
 # In DNS or /etc/hosts make sure that the old and new names resolve correctly. That means that you need entries for controller-# and controller-#-new
 # Update the advertised.listeners and controller.quorum.bootstrap.servers one node at at time. Note that both controllers and brokers need to have the updated controller.quorum.bootstrap.servers
 # In the DNS or /etc/hosts remove the previous or old names: controller-#

Let me know if that works for you. If so, I am willing to review a change to the documentation that describe this process.

> Controllers can't form quorum on restart when hostnames changes with dynamic quorum
> -----------------------------------------------------------------------------------
>
>                 Key: KAFKA-20427
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20427
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Paolo Patierno
>            Priority: Major
>
> It seems that when using the dynamic quorum, it is not possible to recover a KRaft quorum if the controllers change their hostnames due to any DNS infrastructure changes.
> Following the test that I ran.
> I start my Apache Kafka cluster with 3 controllers having hostnames like controller-0, controller-1 and controller-2. I use the /etc/hosts to have them pointing to localhost to run all of them on my laptop. Of course, I format all of them by using the --initial-controllers flag and providing the needed information (including the hostnames for each controller within the list). All the controllers have the controller.quorum.bootstrap.servers configuration properly defined with these hostnames. The controllers start up and they form the quorum.
> I shut down all the controllers and let's assume I am making changes to the DNS infrastructure so that the same controllers will have a different hostname, like controller-0-new, controller-1-new and controller-2-new (I was simulating this locally through the /etc/hosts file. Disabling the old hostnames and enabling the new ones to point to localhost). I change the controller.quorum.bootstrap.servers configuration to use the new hostnames. Then I restart all the controllers again. They are not able to form the quorum anymore.
> During my investigation, the controllers are going to use the metadata topic as the source of truth for checking what are the voters and how to form the quorum. Of course, the metadata topic contains a VoterSet record with the quorum having the old hostnames. So looking at the log it's clear that each controller loads such a record and then tries to connect to the voters with the old hostnames (which are not reachable anymore). It seems there is no mechanism for the controller to use the new controller.quorum.bootstrap.servers with the new hostnames to try starting a leader election for a new quorum.
> {noformat}
> [2026-04-08 14:21:08,500] INFO [RaftManager id=0] Reading KRaft snapshot and log as part of the initialization (org.apache.kafka.raft.KafkaRaftClient)
> [2026-04-08 14:21:08,502] INFO [RaftManager id=0] Loading snapshot (OffsetAndEpoch[offset=0, epoch=0]) since log start offset (0) is greater than the internal listener's next offset (-1) (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine)
> [2026-04-08 14:21:08,507] INFO [RaftManager id=0] Latest kraft.version is KRAFT_VERSION_1 at offset -1 (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine)
> [2026-04-08 14:21:08,508] INFO [RaftManager id=0] Latest set of voters is VoterSet(voters={0=VoterNode(voterKey=ReplicaKey(id=0, directoryId=cCg_WPLrSWaPUCuIhtFHiw), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-0/<unresolved>:9093}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 1=VoterNode(voterKey=ReplicaKey(id=1, directoryId=lIZM5Y0ZQ0GNrFywdVIX6w), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-1/<unresolved>:9094}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 2=VoterNode(voterKey=ReplicaKey(id=2, directoryId=CSUAsFqGSJaBcenHEyS2Yg), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-2/<unresolved>:9095}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1])}) at offset -1 (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine)
> [2026-04-08 14:21:08,511] INFO [RaftManager id=0] Latest kraft.version is KRAFT_VERSION_1 at offset 1 (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine)
> [2026-04-08 14:21:08,512] INFO [RaftManager id=0] Latest set of voters is VoterSet(voters={0=VoterNode(voterKey=ReplicaKey(id=0, directoryId=cCg_WPLrSWaPUCuIhtFHiw), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-0/<unresolved>:9093}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 1=VoterNode(voterKey=ReplicaKey(id=1, directoryId=lIZM5Y0ZQ0GNrFywdVIX6w), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-1/<unresolved>:9094}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 2=VoterNode(voterKey=ReplicaKey(id=2, directoryId=CSUAsFqGSJaBcenHEyS2Yg), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-2/<unresolved>:9095}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1])}) at offset 2 (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine)
> [2026-04-08 14:21:08,522] INFO [RaftManager id=0] Starting voters are VoterSet(voters={0=VoterNode(voterKey=ReplicaKey(id=0, directoryId=cCg_WPLrSWaPUCuIhtFHiw), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-0/<unresolved>:9093}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 1=VoterNode(voterKey=ReplicaKey(id=1, directoryId=lIZM5Y0ZQ0GNrFywdVIX6w), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-1/<unresolved>:9094}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1]), 2=VoterNode(voterKey=ReplicaKey(id=2, directoryId=CSUAsFqGSJaBcenHEyS2Yg), listeners=Endpoints(endpoints={ListenerName(CONTROLLER)=controller-2/<unresolved>:9095}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:1])}) (org.apache.kafka.raft.KafkaRaftClient)
> [2026-04-08 14:21:08,531] INFO [RaftManager id=0] Attempting durable transition to UnattachedState(epoch=2, leaderId=OptionalInt.empty, votedKey=Optional.empty, voters=[0, 1, 2], electionTimeoutMs=1059, highWatermark=Optional.empty) from null (org.apache.kafka.raft.QuorumState)
> [2026-04-08 14:21:08,542] INFO [RaftManager id=0] Completed transition to UnattachedState(epoch=2, leaderId=OptionalInt.empty, votedKey=Optional.empty, voters=[0, 1, 2], electionTimeoutMs=1059, highWatermark=Optional.empty) from null (org.apache.kafka.raft.QuorumState)
> {noformat}
>  
> {noformat}
> [2026-04-08 14:21:13,330] WARN [RaftManager id=0] Error connecting to node controller-2:9095 (id: 2 rack: null isFenced: false) (org.apache.kafka.clients.NetworkClient)java.net.UnknownHostException: controller-2: Name or service not known	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)	at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:125)	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.resolveAddresses(ClusterConnectionStates.java:536)	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511)	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1140)	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:368)	at org.apache.kafka.server.util.InterBrokerSendThread.sendRequests(InterBrokerSendThread.java:146)	at org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:109)	at org.apache.kafka.server.util.InterBrokerSendThread.doWork(InterBrokerSendThread.java:137)	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136){noformat}
> This result looks like a regression compared to using the static quorum (or ZooKeeper in the past). Doing the same test but using the controller.quorum.voters configuration, on restart, the controllers are using such configuration as the source of truth to reach the voters and, taking into account the new hostnames, they are able to form the quorum again.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)