[jira] [Created] (KAFKA-10606) Auto create non-existent topics when fetching metadata for all topics

["Lincong Li (Jira)" <ji...@apache.org>]

Lincong Li created KAFKA-10606:
----------------------------------

             Summary: Auto create non-existent topics when fetching metadata for all topics
                 Key: KAFKA-10606
                 URL: https://issues.apache.org/jira/browse/KAFKA-10606
             Project: Kafka
          Issue Type: Bug
            Reporter: Lincong Li


The "allow auto topic creation" flag is hardcoded to be true for the fetch-all-topic metadata request:

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L37

In the below code, annotation claims that "*This never causes auto-creation*". It it NOT true and auto topic creation still gets triggered under some circumstances. So, this is a bug that needs to be fixed.

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L68


For example, the bug could be manifested in the below situation:

A topic T is being deleted and a request to fetch metadata for all topics gets sent to one broker. The broker reads names of all topics from its metadata cache (shown below).

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1196

Then the broker authorizes all topics and makes sure that they are allowed to be described. Then the broker tries to get metadata for every authorized topic by reading the metadata cache again, once for every topic (show below).

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1240

However, the metadata cache could have been updated while the broker was authorizing all topics and topic T and its metadata no longer exist in the cache since the topic got deleted and metadata update requests eventually got propagated from the controller to all brokers. So, at this point, when the broker tries to get metadata for topic T from its cache, it realizes that it does not exist and the broker tries to "auto create" topic T since the allow-auto-topic-creation flag was set to true in all the fetch-all-topic metadata requests.

I think this bug exists since "*metadataRequest.allowAutoTopicCreation*" was introduced.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)