list of stream or consumer objects per topic

[Mudassir Maredia <sh...@gmail.com>]

I am using kafka api 0.10.

        //Sample code
         List<String> topicsList = new ArrayList<>();
         topicsList.add("topic1");
         topicsList.add("topic2");

          KafkaConsumer consumer = new KafkaConsumer(props);
         consumer.subscribe(topicsList);

Problem:

For each topic, I want to spawn a separate thread who would be handling
data on it. It seems like for that to achieve I have to create multiple
KafkaConsumer. I don't want to do that. Does anyone have any idea how to
achieve that simply.

Previously, in 0.8 version if have used createMessageStreams method which
returns collection of kafkaStream (one for each topic). I want some thing
similar to that.

//0.8 code sample
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.
createMessageStreams(topicCountMap);


Thanks,

Moody

Re: list of stream or consumer objects per topic

["Matthias J. Sax" <ma...@confluent.io>]

Please see my SO answer:
https://stackoverflow.com/questions/39799293/kafka-new-api-0-10-doesnt-provide-a-list-of-stream-and-consumer-objects-per-top/39803689#39803689

-Matthias

On 09/30/2016 12:13 PM, Mudassir Maredia wrote:
> I am using kafka api 0.10.
> 
>         //Sample code
>          List<String> topicsList = new ArrayList<>();
>          topicsList.add("topic1");
>          topicsList.add("topic2");
> 
>           KafkaConsumer consumer = new KafkaConsumer(props);
>          consumer.subscribe(topicsList);
> 
> Problem:
> 
> For each topic, I want to spawn a separate thread who would be handling
> data on it. It seems like for that to achieve I have to create multiple
> KafkaConsumer. I don't want to do that. Does anyone have any idea how to
> achieve that simply.
> 
> Previously, in 0.8 version if have used createMessageStreams method which
> returns collection of kafkaStream (one for each topic). I want some thing
> similar to that.
> 
> //0.8 code sample
> Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.
> createMessageStreams(topicCountMap);
> 
> 
> Thanks,
> 
> Moody
>