[jira] [Commented] (FLINK-13052) Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource

["chaiyongqiang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972243#comment-16972243 ] 

chaiyongqiang commented on FLINK-13052:
---------------------------------------

There're some limits when we support multi source topics in kafkaTableSourceSinkFactoryBase. It's like :

When we assign the startup mode as specific-offsets, it's difficult to assign the start-offset for each topic partition because of the way to set the connector configs.  As the following codes show, when we assign several topics like MyTable, MyTable1,MyTable2 , it's difficult to distinguish the partition and offset belong to wich topic. 

{code:java}
final Map<String, String> props2 = new HashMap<>();
		props2.put("connector.property-version", "1");
		props2.put("connector.type", "kafka");
		props2.put("connector.version", "0.11");
		props2.put("connector.topic", "MyTable");
		props2.put("connector.startup-mode", "specific-offsets");
		props2.put("connector.specific-offsets.0.partition", "0");
		props2.put("connector.specific-offsets.0.offset", "42");
		props2.put("connector.specific-offsets.1.partition", "1");
{code}

But fortunately， setting a fixed topic-partition start offset when we start an app with table API is rare. For now,  i suggest ignoring this situation.  My solution is supporting multi-topic for other kind of start-up mode like GROUP_OFFSETS, EARLIEST,LATEST,  for SPECIFIC_OFFSETS mode  when using Table API like kafkaTableSourceSinkFactoryBase with several topics we use the GROUP_OFFSETS instead. 

> Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource
> -----------------------------------------------------------------------------------------
>
>                 Key: FLINK-13052
>                 URL: https://issues.apache.org/jira/browse/FLINK-13052
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka, Table SQL / API
>    Affects Versions: 1.8.0
>            Reporter: chaiyongqiang
>            Priority: Critical
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)