[jira] [Created] (KAFKA-20989) KRaft dual-write does not clean up ZooKeeper config and ACL notification znodes

["Youssef Ismail (Jira)" <ji...@apache.org>]

Youssef Ismail created KAFKA-20989:
--------------------------------------

             Summary: KRaft dual-write does not clean up ZooKeeper config and ACL notification znodes
                 Key: KAFKA-20989
                 URL: https://issues.apache.org/jira/browse/KAFKA-20989
             Project: Kafka
          Issue Type: Bug
          Components: kraft
    Affects Versions: 3.9.1
            Reporter: Youssef Ismail


During ZooKeeper-to-KRaft migration, the KRaft controller creates persistent sequential notifications for [config changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkConfigMigrationClient.scala#L352-L357] and [ACL changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkAclMigrationClient.scala#L41-L44].

These nodes are normally deleted after 15 minutes by [{{ZkNodeChangeNotificationListener}}|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala#L125-L140], which is started by [ZooKeeper-mode brokers|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/KafkaServer.scala#L631-L633].

Once all brokers enter KRaft mode in dual-write phase no cleanup listener remains, while the [migration controller continues dual-writing|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/ControllerServer.scala#L284-L314]. Consequently, {{{}/config/changes{}}}, {{{}/kafka-acl-changes{}}}, and {{/kafka-acl-extended-changes}} keep growing without deletion.

Normally if the migration happens quickly this shouldn't be an issue  However we want to let our clusters bake for some time in this phase before finalizing the migration which causes the backlog.

This can increase ZooKeeper load and eventually make {{getChildren}} exceed {{{}jute.maxbuffer{}}}, potentially impairing rollback.

I think the simplest solution is to have the migration controller run an equivalent cleanup while migration mode is enabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)