You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Gaurav Narula (Jira)" <ji...@apache.org> on 2026/09/22 14:50:00 UTC
[jira] [Updated] (KAFKA-21142) KRaftMigrationZkWriter may fail to create TopicPartitionStateZnode during a crash fault
[ https://issues.apache.org/jira/browse/KAFKA-21142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gaurav Narula updated KAFKA-21142:
----------------------------------
Description:
Kafka quorum controller leader may fail to correctly update state in ZK when it crashes in the following scenario
*Setup*
Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is acting as the coordinator and owning writes to ZK via KRaftMigrationDriver. Also consider the brokers to not be bounced in KRaft mode.
*Failure Scenario*
1. Let's say a client sends a request to increase the topic-partition count of topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by {{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum leader crashes at this point before any updates to ZK._
3. Eventually, a new quorum leader is elected and it attempts to synchronise the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in {{SyncKRaftMetadataEvent}} and eventually the invocation of {{KRaftMigrationZkWriter#handleSnapshot}} followed by {{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually, [newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217] and [changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224] are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being updated but the handler incorrectly calls {{migrationClient.topicClient().updateTopicPartitions()}} instead of {{migrationClient.topicClient().createTopicPartitions}} [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a {{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to be retried.
8. This time around, {{newPartitions}} is not populated because {{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a placeholder record [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated
was:
Kafka quorum controller leader may fail to correctly update state in ZK when it crashes in the following scenario
*Setup*
Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is acting as the coordinator and owning writes to ZK via KRaftMigrationDriver. Also consider the brokers to not be bounced in KRaft mode.
*Failure Scenario*
1. Let's say a client sends a request to increase the topic-partition count of topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by {{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum leader crashes after this operation._
3. Eventually, a new quorum leader is elected and it attempts to synchronise the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in {{SyncKRaftMetadataEvent}} and eventually the invocation of {{KRaftMigrationZkWriter#handleSnapshot}} followed by {{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually, [newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217] and [changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224] are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being updated but the handler incorrectly calls {{migrationClient.topicClient().updateTopicPartitions()}} instead of {{migrationClient.topicClient().createTopicPartitions}} [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a {{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to be retried.
8. This time around, {{newPartitions}} is not populated because {{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a placeholder record [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated
> KRaftMigrationZkWriter may fail to create TopicPartitionStateZnode during a crash fault
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-21142
> URL: https://issues.apache.org/jira/browse/KAFKA-21142
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.9.2
> Reporter: Gaurav Narula
> Assignee: Gaurav Narula
> Priority: Major
>
> Kafka quorum controller leader may fail to correctly update state in ZK when it crashes in the following scenario
>
> *Setup*
> Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is acting as the coordinator and owning writes to ZK via KRaftMigrationDriver. Also consider the brokers to not be bounced in KRaft mode.
> *Failure Scenario*
>
> 1. Let's say a client sends a request to increase the topic-partition count of topic {{T1}} from {{2}} to {{3}}
> 2. After the request is processed by the quorum leader, the delta is propagated to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by {{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum leader crashes at this point before any updates to ZK._
> 3. Eventually, a new quorum leader is elected and it attempts to synchronise the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in {{SyncKRaftMetadataEvent}} and eventually the invocation of {{KRaftMigrationZkWriter#handleSnapshot}} followed by {{KRaftMigrationZkWriter#handleTopicsSnapshot}}
> 4. Eventually, [newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217] and [changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224] are populated.
> 5. {{changedTopics}} results in the {{TopicZnode}} being updated with information about the new replica
> 6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being updated but the handler incorrectly calls {{migrationClient.topicClient().updateTopicPartitions()}} instead of {{migrationClient.topicClient().createTopicPartitions}} [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
> 7. This results in a SetData on a ZNode which doesn't exist and ultimately a {{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to be retried.
> 8. This time around, {{newPartitions}} is not populated because {{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a placeholder record [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
> 9. Ultimately, the event is processed with ZK state not having been updated
--
This message was sent by Atlassian Jira
(v8.20.10#820010)