You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rahul Mittal <ra...@gmail.com> on 2014/09/12 11:49:55 UTC
Kafka High Level Consumer
Hi ,
Is there a way in kafka to read data from all topics, from a consumer group
without specifying topics in a dynamic way.
That is if new topics are created on kafka brokers the consumer group
should figure it out and start reading from the new topic as well without
explicitly defining new topic for consumer.
--
*With Regards *
*RAHUL MITTAL*
Re: Kafka High Level Consumer
Posted by Rahul Mittal <ra...@gmail.com>.
Thanks Joe Stein
This worked :)
On Fri, Sep 12, 2014 at 3:19 PM, Rahul Mittal <ra...@gmail.com>
wrote:
> Hi ,
> Is there a way in kafka to read data from all topics, from a consumer
> group without specifying topics in a dynamic way.
> That is if new topics are created on kafka brokers the consumer group
> should figure it out and start reading from the new topic as well without
> explicitly defining new topic for consumer.
>
> --
> *With Regards *
> *RAHUL MITTAL*
>
>
--
*With Regards *
*RAHUL MITTAL*
*FINAL YEAR*
*CSE*
*MNNIT ALLAHABAD*
Re: Kafka High Level Consumer
Posted by Joe Stein <jo...@stealth.ly>.
You want to use the createMessageStreamsByFilter and pass in a WhiteList
with a regex that would include everything you want... here is e.g. how to
use that
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L196
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Sep 12, 2014 at 5:49 AM, Rahul Mittal <ra...@gmail.com>
wrote:
> Hi ,
> Is there a way in kafka to read data from all topics, from a consumer group
> without specifying topics in a dynamic way.
> That is if new topics are created on kafka brokers the consumer group
> should figure it out and start reading from the new topic as well without
> explicitly defining new topic for consumer.
>
> --
> *With Regards *
> *RAHUL MITTAL*
>