Multiple topics to one consumer

[복영빈 <bo...@gmail.com>]

Hi!
I am quite new to Kafka and pub-sub systems.

I am trying to make a single consumer to consume messages from multiple
topics.

Although I understand that one topic can be sent to multiple consumers via
partitions,

I could not find the part in the documentation that specifies that a single
consumer can consume data from multiple topics.

My understanding is that one consumer holds one thread until it is closed,
by looping indefinitely on while(it.hasNext())).

So I assumed that to read from multiple topics, I would have to create
multiple consumers that each read from a single topic.

Am I right in that a single consumer cannot read from multiple topics?

If not, how would I achieve this?

Thanks in advance!

Re: Multiple topics to one consumer

[Alex Loddengaard <al...@confluent.io>]

Hi there,

One consumer can indeed consume from multiple topics (and multiple
partitions). For example, see here:

http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.List)

Then, in your poll() loop, you can get the topic and partition from the
returned ConsumerRecord.

I hope this helps.

Alex

On Tue, Mar 8, 2016 at 1:12 AM, 복영빈 <bo...@gmail.com> wrote:

> Hi!
> I am quite new to Kafka and pub-sub systems.
>
> I am trying to make a single consumer to consume messages from multiple
> topics.
>
> Although I understand that one topic can be sent to multiple consumers via
> partitions,
>
> I could not find the part in the documentation that specifies that a single
> consumer can consume data from multiple topics.
>
> My understanding is that one consumer holds one thread until it is closed,
> by looping indefinitely on while(it.hasNext())).
>
> So I assumed that to read from multiple topics, I would have to create
> multiple consumers that each read from a single topic.
>
> Am I right in that a single consumer cannot read from multiple topics?
>
> If not, how would I achieve this?
>
> Thanks in advance!
>