You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Nicolas Perrin (Jira)" <ji...@apache.org> on 2024/04/22 14:50:00 UTC

[jira] [Created] (FLINK-35210) Give the option to set automatically the parallelism of the KafkaSource to the number of kafka partitions

Nicolas Perrin created FLINK-35210:
--------------------------------------

             Summary: Give the option to set automatically the parallelism of the KafkaSource to the number of kafka partitions
                 Key: FLINK-35210
                 URL: https://issues.apache.org/jira/browse/FLINK-35210
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kafka
            Reporter: Nicolas Perrin


Currently the setting of the `KafkaSource` Flink's operator parallelism needs to be manually chosen which can leads to highly skewed tasks if the developer doesn't do this job.

To avoid this issue, I propose to:
-  retrieve dynamically the number of partitions of the topic using `KafkaConsumer.
partitionsFor(topic).size()`,
- set the parallelism of the stream built from the source based on this value.

 This way there won't be any idle tasks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)