multiple consumers for a same topic in multiple copies

[Patrick Viet <pa...@gmail.com>]

Hi,

I've done some searching and can't find much on this topic.
Here is what I'm looking at:

So a usual usage of kafka is to push messages in a topic, and then have
one/several consumers consume it (one per partition), in a "unique" way: a
message is consumed only once, and if it's consumed multiple times it's a
mistake, which we try to avoid.

What if I'd like several consumers to get the same messages?
Here is a practical example:

- Some app produces user action messages
- One entity (say, marketing) would like to consume them to do analysis
- Another entity (say, BI) would like to consume them too, to do some other
analysis

So effectively both would get the full (duplicate) messages

It seems that in the current high level consumer model, the messages only
go to one entity at a time, commits to zookeeper where it's at every X
messages read from partition Y, etc. My understanding is that if I want to
be able to read the messages in any other way, I'd have to make my own
consumer based on raw info from kafka and zookeeper.

But maybe I'm wrong and it's totally possible in the current model to have
several consumers subscribe independently on a same topic, and consume then
with their own progression markers. If yes, a pointer to any documentation
/ examples / discussions would be greatly appreciated

-- 
Best regards,
Patrick Viet

Re: multiple consumers for a same topic in multiple copies

[Luca Pertile <pe...@gmail.com>]

Hi Patrick,

As far as I understand you can achieve that level of isolation by assigning
the BI/Marketing consumers to distinct  consumer *groups *for the topic/s,
that you can for instance name accordingly. Within each consumer group you
can then have a Marketing and a BI consumer that will be able to commit
their offset in kafka independently as they move along the topic.

Luca

On 31 Oct 2016 16:42, "Patrick Viet" <pa...@gmail.com> wrote:

> Hi,
>
> I've done some searching and can't find much on this topic.
> Here is what I'm looking at:
>
> So a usual usage of kafka is to push messages in a topic, and then have
> one/several consumers consume it (one per partition), in a "unique" way: a
> message is consumed only once, and if it's consumed multiple times it's a
> mistake, which we try to avoid.
>
> What if I'd like several consumers to get the same messages?
> Here is a practical example:
>
> - Some app produces user action messages
> - One entity (say, marketing) would like to consume them to do analysis
> - Another entity (say, BI) would like to consume them too, to do some other
> analysis
>
> So effectively both would get the full (duplicate) messages
>
> It seems that in the current high level consumer model, the messages only
> go to one entity at a time, commits to zookeeper where it's at every X
> messages read from partition Y, etc. My understanding is that if I want to
> be able to read the messages in any other way, I'd have to make my own
> consumer based on raw info from kafka and zookeeper.
>
> But maybe I'm wrong and it's totally possible in the current model to have
> several consumers subscribe independently on a same topic, and consume then
> with their own progression markers. If yes, a pointer to any documentation
> / examples / discussions would be greatly appreciated
>
> --
> Best regards,
> Patrick Viet
>

Re: multiple consumers for a same topic in multiple copies

[Patrick Viet <pa...@gmail.com>]

Yes! This was what I was looking for:

"
If all the consumer instances have the same consumer group, then the
records will effectively be load balanced over the consumer instances.
If all the consumer instances have different consumer groups, then each
record will be broadcast to all the consumer processes.
"

I can't believe I missed it the first time.
Thanks again

--Patrick


On Mon, Oct 31, 2016 at 12:46 PM, Samuel Taylor <st...@square-root.com>
wrote:
>
> Hi Patrick,
>
> Can you use separate consumer groups to accomplish what you're looking
for?
>
> Scroll to the "consumers" section on this document for a more detailed
> description:
>
> https://kafka.apache.org/documentation.html#introduction
>
> On Mon, Oct 31, 2016 at 11:42 AM, Patrick Viet <pa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I've done some searching and can't find much on this topic.
> > Here is what I'm looking at:
> >
> > So a usual usage of kafka is to push messages in a topic, and then have
> > one/several consumers consume it (one per partition), in a "unique"
way: a
> > message is consumed only once, and if it's consumed multiple times it's
a
> > mistake, which we try to avoid.
> >
> > What if I'd like several consumers to get the same messages?
> > Here is a practical example:
> >
> > - Some app produces user action messages
> > - One entity (say, marketing) would like to consume them to do analysis
> > - Another entity (say, BI) would like to consume them too, to do some
other
> > analysis
> >
> > So effectively both would get the full (duplicate) messages
> >
> > It seems that in the current high level consumer model, the messages
only
> > go to one entity at a time, commits to zookeeper where it's at every X
> > messages read from partition Y, etc. My understanding is that if I want
to
> > be able to read the messages in any other way, I'd have to make my own
> > consumer based on raw info from kafka and zookeeper.
> >
> > But maybe I'm wrong and it's totally possible in the current model to
have
> > several consumers subscribe independently on a same topic, and consume
then
> > with their own progression markers. If yes, a pointer to any
documentation
> > / examples / discussions would be greatly appreciated
> >
> > --
> > Best regards,
> > Patrick Viet
> >
>
>
>
> --
> *Samuel Taylor*
> Data Science
>
> *Square Root, Inc. <http://square-root.com/>*
> <http://square-root.com/>Square-Root.com <http://square-root.com/>

Re: multiple consumers for a same topic in multiple copies

[Samuel Taylor <st...@square-root.com>]

Hi Patrick,

Can you use separate consumer groups to accomplish what you're looking for?

Scroll to the "consumers" section on this document for a more detailed
description:

https://kafka.apache.org/documentation.html#introduction

On Mon, Oct 31, 2016 at 11:42 AM, Patrick Viet <pa...@gmail.com>
wrote:

> Hi,
>
> I've done some searching and can't find much on this topic.
> Here is what I'm looking at:
>
> So a usual usage of kafka is to push messages in a topic, and then have
> one/several consumers consume it (one per partition), in a "unique" way: a
> message is consumed only once, and if it's consumed multiple times it's a
> mistake, which we try to avoid.
>
> What if I'd like several consumers to get the same messages?
> Here is a practical example:
>
> - Some app produces user action messages
> - One entity (say, marketing) would like to consume them to do analysis
> - Another entity (say, BI) would like to consume them too, to do some other
> analysis
>
> So effectively both would get the full (duplicate) messages
>
> It seems that in the current high level consumer model, the messages only
> go to one entity at a time, commits to zookeeper where it's at every X
> messages read from partition Y, etc. My understanding is that if I want to
> be able to read the messages in any other way, I'd have to make my own
> consumer based on raw info from kafka and zookeeper.
>
> But maybe I'm wrong and it's totally possible in the current model to have
> several consumers subscribe independently on a same topic, and consume then
> with their own progression markers. If yes, a pointer to any documentation
> / examples / discussions would be greatly appreciated
>
> --
> Best regards,
> Patrick Viet
>



-- 
*Samuel Taylor*
Data Science

*Square Root, Inc. <http://square-root.com/>*
<http://square-root.com/>Square-Root.com <http://square-root.com/>

Re: multiple consumers for a same topic in multiple copies

[Hans Jespersen <ha...@confluent.io>]

Just make sure they are not in the same consumer group by creating a unique
value for group.id for each independent consumer.

/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * hans@confluent.io (650)924-2670
 */

On Mon, Oct 31, 2016 at 9:42 AM, Patrick Viet <pa...@gmail.com>
wrote:

> Hi,
>
> I've done some searching and can't find much on this topic.
> Here is what I'm looking at:
>
> So a usual usage of kafka is to push messages in a topic, and then have
> one/several consumers consume it (one per partition), in a "unique" way: a
> message is consumed only once, and if it's consumed multiple times it's a
> mistake, which we try to avoid.
>
> What if I'd like several consumers to get the same messages?
> Here is a practical example:
>
> - Some app produces user action messages
> - One entity (say, marketing) would like to consume them to do analysis
> - Another entity (say, BI) would like to consume them too, to do some other
> analysis
>
> So effectively both would get the full (duplicate) messages
>
> It seems that in the current high level consumer model, the messages only
> go to one entity at a time, commits to zookeeper where it's at every X
> messages read from partition Y, etc. My understanding is that if I want to
> be able to read the messages in any other way, I'd have to make my own
> consumer based on raw info from kafka and zookeeper.
>
> But maybe I'm wrong and it's totally possible in the current model to have
> several consumers subscribe independently on a same topic, and consume then
> with their own progression markers. If yes, a pointer to any documentation
> / examples / discussions would be greatly appreciated
>
> --
> Best regards,
> Patrick Viet
>