You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Amikam Snir <am...@gmail.com> on 2014/05/12 08:59:40 UTC

How to consume data from multiple Kafka topics?

Hi all,

Do you know code (Spout-code), which consume from multiple topics for
example using wildcard against the topic name?



The topics at my application are created dynamically. Should I just use
dedicated Kafka-Spout for each topic and somehow updated the topology at
run-time?





Thanks in advance,

Amikam.

Re: How to consume data from multiple Kafka topics?

Posted by Joe Stein <jo...@stealth.ly>.
In Kafka you can use the WhiteList and BlackList topic filters (to regex
topics that match or do not match respectively).  The console consumer is a
nice example how to-do this with the high level consumer
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L137then
when creating a stream that you can iterate over using that filterSpec
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L196and
you are good to go.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Mon, May 12, 2014 at 3:10 AM, David Miller
<da...@m-square.com.au>wrote:

>
> The kafka "high level" consumer supports topic filtering for doing
> wildcards against topics.
> The storm-kafka spout uses the simple consumer which doesnt have these
> features (but has others needed for reliable messaging, offset management)
>
> you could write your own high level consumer spout which could support
> this feature at the expense of some message reliability.
>
> Look at TopicFilter
> https://kafka.apache.org/08/api.html
>
>
>
>
>
>
> On Mon, May 12, 2014 at 4:59 PM, Amikam Snir <am...@gmail.com> wrote:
>
>> Hi all,
>>
>> Do you know code (Spout-code), which consume from multiple topics for
>> example using wildcard against the topic name?
>>
>>
>>
>> The topics at my application are created dynamically. Should I just use
>> dedicated Kafka-Spout for each topic and somehow updated the topology at
>> run-time?
>>
>>
>>
>>
>>
>> Thanks in advance,
>>
>> Amikam.
>>
>
>

Re: How to consume data from multiple Kafka topics?

Posted by David Miller <da...@m-square.com.au>.
The kafka "high level" consumer supports topic filtering for doing
wildcards against topics.
The storm-kafka spout uses the simple consumer which doesnt have these
features (but has others needed for reliable messaging, offset management)

you could write your own high level consumer spout which could support this
feature at the expense of some message reliability.

Look at TopicFilter
https://kafka.apache.org/08/api.html






On Mon, May 12, 2014 at 4:59 PM, Amikam Snir <am...@gmail.com> wrote:

> Hi all,
>
> Do you know code (Spout-code), which consume from multiple topics for
> example using wildcard against the topic name?
>
>
>
> The topics at my application are created dynamically. Should I just use
> dedicated Kafka-Spout for each topic and somehow updated the topology at
> run-time?
>
>
>
>
>
> Thanks in advance,
>
> Amikam.
>