[jira] [Updated] (FLINK-36780) Kafka source disable partition discovery unexpectedly

["Arvid Heise (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-36780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvid Heise updated FLINK-36780:
--------------------------------
    Issue Type: Bug  (was: Improvement)

> Kafka source disable partition discovery unexpectedly
> -----------------------------------------------------
>
>                 Key: FLINK-36780
>                 URL: https://issues.apache.org/jira/browse/FLINK-36780
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>    Affects Versions: kafka-4.0.0, kafka-3.2.0, kafka-3.3.0, kafka-3.4.0
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kafka-4.0.0
>
>
> Currently Kafka source enables partition discovery. This is set by {{partition.discovery.interval.ms}}, aka {{KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS}}. The default value is 5 minutes, which is equal to the default value of {{metadata.max.age.ms}} in Kafka.
> However, it's disabled by default unexpectedly in the source builder ([code|https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L476-L480]). The intention I believe was to only disable for bounded source.
> We need a fix that is able to keep the default partition discovery. This could cause data loss after Kafka retention if the new partitions are not consumed silently.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)