[jira] [Assigned] (FLINK-39238) Support watermark alignment in flink-connector-kafka in dynamic Kafka source mode

["Bowen Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bowen Li reassigned FLINK-39238:
--------------------------------

    Assignee: Xin Gao

> Support watermark alignment in flink-connector-kafka in dynamic Kafka source mode
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-39238
>                 URL: https://issues.apache.org/jira/browse/FLINK-39238
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>            Reporter: Xin Gao
>            Assignee: Xin Gao
>            Priority: Critical
>              Labels: pull-request-available
>
> Flink source ingestion is with watermark alignment config like `scan.watermark.alignment.max-drift`. The operator would then control
>  * checkWatermarkAlignment for whole reader
>  * checkSplitWatermarkAlignment for the split reader
> In dynamic Kafka source mode, the split reader pause/resume is missed (`pauseOrResumeSplits` not implemented) and the exceptions might raise once such config enabled.
> This feature is important for workflows like ingesting data from Kafka to Data Lake. Missing the alignment could create far more number of small files once the ingestion progress diverges and runs our of controls.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)