[jira] [Created] (KAFKA-21142) KRaftMigrationZkWriter may fail to create TopicPartitionStateZnode during a crash fault

["Gaurav Narula (Jira)" <ji...@apache.org>]

Gaurav Narula created KAFKA-21142:
-------------------------------------

             Summary: KRaftMigrationZkWriter may fail to create TopicPartitionStateZnode during a crash fault
                 Key: KAFKA-21142
                 URL: https://issues.apache.org/jira/browse/KAFKA-21142
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 3.9.2
            Reporter: Gaurav Narula
            Assignee: Gaurav Narula


Kafka quorum controller leader may fail to correctly update state in ZK when it crashes in the following scenario
 
*Setup*

Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is acting as the coordinator and owning writes to ZK via KRaftMigrationDriver. Also consider the brokers to not be bounced in KRaft mode.

*Failure Scenario*
 
1. Let's say a client sends a request to increase the topic-partition count of topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by {{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum leader crashes after this operation._
3. Eventually, a new quorum leader is elected and it attempts to synchronise the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in {{SyncKRaftMetadataEvent}} and eventually the invocation of {{KRaftMigrationZkWriter#handleSnapshot}} followed by {{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually, [newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217] and [changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224] are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being updated but the handler incorrectly calls {{migrationClient.topicClient().updateTopicPartitions()}} instead of {{migrationClient.topicClient().createTopicPartitions}} [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a {{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to be retried.
8. This time around, {{newPartitions}} is not populated because {{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a placeholder record [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated



--
This message was sent by Atlassian Jira
(v8.20.10#820010)