[jira] [Created] (FLINK-37864) Extend DynamicKafkaSource to allow for cluster-specific OffsetsInitializer

["Joey Pereira (Jira)" <ji...@apache.org>]

Joey Pereira created FLINK-37864:
------------------------------------

             Summary: Extend DynamicKafkaSource to allow for cluster-specific OffsetsInitializer 
                 Key: FLINK-37864
                 URL: https://issues.apache.org/jira/browse/FLINK-37864
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kafka
    Affects Versions: 2.0.0
            Reporter: Joey Pereira


At the moment, the DynamicKafkaSource cannot have OffsetsInitializers that vary per-cluster but only per-topic and partition provided the parameterization of {{org.apache.kafka.common.TopicPartition}}.

For more complex use-cases with consumption from multiple Kafka sources, it would be ideal to allow for offset customization per-cluster.

One way this can be added while keeping things simple would be to:
- Add {{setStartingOffsets}} and {{setBounded}} with the arguments {{Map<String, OffsetsInitializer>}}
- Plumb that map as a seperate property through to the {{DynamicKafkaSourceEnumerator}} and select the appropriate one during {{createEnumeratorWithAssignedTopicPartitions}} 

Additionally, we'll want to:
- Include adjustments to a Python interface FLINK-37863
- Adjust the documentation

(cc [~shuyichen])



--
This message was sent by Atlassian Jira
(v8.20.10#820010)