[jira] [Created] (FLINK-41100) DynamicKafkaSource shares one split output between partitions whose split ids collide

["Martijn Visser (Jira)" <ji...@apache.org>]

Martijn Visser created FLINK-41100:
--------------------------------------

             Summary: DynamicKafkaSource shares one split output between partitions whose split ids collide
                 Key: FLINK-41100
                 URL: https://issues.apache.org/jira/browse/FLINK-41100
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
    Affects Versions: kafka-5.0.0, kafka-4.0.1, kafka-3.2.0
            Reporter: Martijn Visser
            Assignee: Martijn Visser


{{DynamicKafkaSourceSplit#splitId}} is {{<cluster>-<topic>-<partition>}}, so cluster {{a}} with topic {{b-c}} and cluster {{a-b}} with topic {{c}} both get {{a-b-c-0}}. The sub-readers of {{DynamicKafkaSourceReader}} emit into the same {{ReaderOutput}} under that id ({{KafkaPartitionSplitReaderWrapper}} prefixes the cluster id), and {{createOutputForSplit}} returns the existing output for an id it already knows. When one reader holds both partitions, they share one split-local watermark generator, and when one of them finishes, {{SourceReaderBase}} releases the output the other one still writes to. From then on that partition no longer holds back the combined watermark, so its records can be late. Watermark alignment pauses and resumes both partitions together for the same reason.

The split ids used towards Flink should be unique per physical partition. FLINK-41088 fixes the same collision in the enumerator's recovery paths.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)