[jira] [Updated] (FLINK-40469) DynamicKafkaSource can lose splits when checkpointing during recovery reassignment

["Martijn Visser (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser updated FLINK-40469:
-----------------------------------
    Release Note: DynamicKafkaSource no longer loses partitions when a checkpoint completes while restored splits wait for reassignment. The enumerator state version changes, so a checkpoint taken with 5.1.0 cannot be restored with 5.0.0.

> DynamicKafkaSource can lose splits when checkpointing during recovery reassignment
> ----------------------------------------------------------------------------------
>
>                 Key: FLINK-40469
>                 URL: https://issues.apache.org/jira/browse/FLINK-40469
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>            Reporter: Aleksandr Savonin
>            Assignee: Aleksandr Savonin
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kafka-5.1.0
>
>
> After recovery, readers report their restored splits to DynamicKafkaSourceEnumerator and temporarily hold no splits while metadata discovery is pending.
> These reports are kept only in memory. If a checkpoint completes during this window and the job later restores from it, the reader state is empty while the sub-enumerator still marks the splits as assigned. The splits are therefore never reassigned, causing silent data loss.
> Persist pending reader-reported splits in DynamicKafkaSourceEnumState and restore them through the existing reassignment flow, preserving their checkpointed offsets.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)