You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by chandresh pancholi <ch...@gmail.com> on 2018/04/11 12:33:51 UTC

How to add new Kafka topic to consumer

Hi,

Is there a way to add Kafka topic dynamically to stream?
For example there were Kafka topic named foo1, foo2, foo3 and task manager
will start consuming events from all 3 topics. Now I create another two
topic foo4 & bar1 in Kafka.

How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?

Thanks

-- 
Chandresh Pancholi
Senior Software Engineer
Flipkart.com
Email-id:chandresh.pancholi@flipkart.com
Contact:08951803660

Re: How to add new Kafka topic to consumer

Posted by Alexander Smirnov <al...@gmail.com>.

this feature has been implemented in 1.4.0, take a look at

https://issues.apache.org/jira/browse/FLINK-4022
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html#kafka-consumers-topic-and-partition-discovery


On Wed, Apr 11, 2018 at 3:33 PM chandresh pancholi <
chandreshpancholi007@gmail.com> wrote:

> Hi,
>
> Is there a way to add Kafka topic dynamically to stream?
> For example there were Kafka topic named foo1, foo2, foo3 and task manager
> will start consuming events from all 3 topics. Now I create another two
> topic foo4 & bar1 in Kafka.
>
> How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?
>
> Thanks
>
> --
> Chandresh Pancholi
> Senior Software Engineer
> Flipkart.com
> Email-id:chandresh.pancholi@flipkart.com
> Contact:08951803660
>

Re: How to add new Kafka topic to consumer

Posted by Chesnay Schepler <ch...@apache.org>.

In 1.3 and below I believe this is not possible.

For 1.4 and above, please see 
https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html#partition-discovery.

On 11.04.2018 14:33, chandresh pancholi wrote:
> Hi,
>
> Is there a way to add Kafka topic dynamically to stream?
> For example there were Kafka topic named foo1, foo2, foo3 and task 
> manager will start consuming events from all 3 topics. Now I create 
> another two topic foo4 & bar1 in Kafka.
>
> How will FlinkKafkaConsumer would read events from foo4 & bar1 topic?
>
> Thanks
>
> -- 
> Chandresh Pancholi
> Senior Software Engineer
> Flipkart.com
> Email-id:chandresh.pancholi@flipkart.com 
> <ma...@flipkart.com>
> Contact:08951803660