Re: [DISCUSS] Multi-topics consuming from KafkaTableSource

[leo <le...@qq.com>]

Hi all,


Thanks for your involvement. The benefits can be summarized here for a 
multi-topics consuming via KafkaTableSource,

First, It boosts developing and managing a source with so many topics, 
especially when the schema of topics is the same.

Second, It is aligned with existed FlinkKafkaConsumer and it is also 
easy to achieve this goal just leveraging on the existed APIs.

I'll open a VOTE for this feature now and please join this.


Best,

Leo


On 11/27/19 3:12 PM, Gyula Fóra wrote:
> Hi all,
>
> I think the same reasoning applies here as with the regular Kafka consumer
> sources.
>
> Those support multiple topics, even topic patterns assuming the same schema.
>
> Some times you have so many topics that you can't create a source for each
> or as with the pattern the topics might be changing over time.
>
> Cheers
> Gyula
>
> On Wed, Nov 27, 2019, 07:36 Jingsong Li <ji...@gmail.com> wrote:
>
>> Hi leo:
>>
>> Can you share why KafkaSource supports multiple topics?
>> Why can't we use multiple sources and then union them?
>>
>> Best,
>> Jingsong Lee
>>
>> On Wed, Nov 27, 2019 at 8:19 AM leo <le...@qq.com> wrote:
>>
>>> Hi,
>>>
>>> As Flink does not own the data, so it can not forbid this. It's the
>>> responsibility of user to deserialize messages from these topics with
>>> custom schema deser.
>>>
>>>
>>> On 11/26/19 10:55 AM, Jun Zhang wrote:
>>>> If the data formats of the Multi-topics are different, what do you
>>>> plan to do?
>>>>
>>>> On 11/26/2019 08:53，leo<le...@qq.com>
>>>> <ma...@qq.com> wrote：
>>>>
>>>>
>>>>      Hi, all.
>>>>
>>>>
>>>>      I want to propose an improvement of KafkaTableSource which can
>>>>      consume
>>>>      multiple topics at the same time.
>>>>
>>>>      Here is proposal,
>>>>      https://issues.apache.org/jira/browse/FLINK-14729
>>>>
>>>>
>>>>      Best,
>>>>
>>>>      Leo
>>>>
>>>>
>>
>> --
>> Best, Jingsong Lee
>>