[jira] [Created] (FLINK-32020) Enable Dynamic Partition Discovery by Default in Kafka Source based on FLIP-288

["Hongshun Wang (Jira)" <ji...@apache.org>]

Hongshun Wang created FLINK-32020:
-------------------------------------

             Summary: Enable Dynamic Partition Discovery by Default in Kafka Source based on FLIP-288
                 Key: FLINK-32020
                 URL: https://issues.apache.org/jira/browse/FLINK-32020
             Project: Flink
          Issue Type: Sub-task
            Reporter: Hongshun Wang


As described in [FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source], dynamic partition discovery is disabled by default, and users have to specify the interval of discovery in order to turn it on.

This subtask is to enable Dynamic Partition Discovery by Default in Kafka Source.

Partition discovery is performed on the KafkaSourceEnumerator, which asynchronously fetches topic metadata from the Kafka cluster and checks if there are any new topics and partitions. This should not cause performance issues on the Flink side.

On the Kafka broker side, partition discovery sends a MetadataRequest to the Kafka broker to fetch topic information. Considering that the Kafka broker has its metadata cache and the default request frequency is relatively low (once every 30 seconds), this is not a heavy operation, and the broker's performance will not be significantly affected.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)