kafka 1:N message delivery pattern

[Chandra R <my...@gmail.com>]

Hello All,

I am trying to understand the Kafka 1:N pattern.
I am trying to understand it in terms of a twitter like app.

For every user there will be 1 Producer. Each follower will be in its own
consumer group and there will be a Broker which will connect Producer to
Consumer groups.

Am I getting it right ?

Thanks. Any advice would be appreciated.

regards

Chandra R

Re: kafka 1:N message delivery pattern

[Chandra R <my...@gmail.com>]

Thank you all,

I get it now.
I was trying thinking in terms of RabbitMQ's fanout exchange.

So from what I understand it will be difficult to implement if number of
users are in millions (or in our case grows to millions),
as we need a topic for each user.

I wish there was an option thought :)

reagrds,

Chandra



On Fri, Sep 9, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:

> Chandra,
>
> Let me see if I understand what you want to do. There is one topic per user
> (which stores all tweets posted by this user). Each follower consumes the
> topics corresponding to the users he follows. Conceptually, this works. The
> question is the number of topics. Currently, Kafka is not suitable for
> millions of topics.
>
> Jun
>
> On Fri, Sep 9, 2011 at 1:40 PM, Chandra R <my...@gmail.com> wrote:
>
> > Hello All,
> >
> > I am trying to understand the Kafka 1:N pattern.
> > I am trying to understand it in terms of a twitter like app.
> >
> > For every user there will be 1 Producer. Each follower will be in its own
> > consumer group and there will be a Broker which will connect Producer to
> > Consumer groups.
> >
> > Am I getting it right ?
> >
> > Thanks. Any advice would be appreciated.
> >
> > regards
> >
> > Chandra R
> >
>

Re: kafka 1:N message delivery pattern

[Jun Rao <ju...@gmail.com>]

Chandra,

Let me see if I understand what you want to do. There is one topic per user
(which stores all tweets posted by this user). Each follower consumes the
topics corresponding to the users he follows. Conceptually, this works. The
question is the number of topics. Currently, Kafka is not suitable for
millions of topics.

Jun

On Fri, Sep 9, 2011 at 1:40 PM, Chandra R <my...@gmail.com> wrote:

> Hello All,
>
> I am trying to understand the Kafka 1:N pattern.
> I am trying to understand it in terms of a twitter like app.
>
> For every user there will be 1 Producer. Each follower will be in its own
> consumer group and there will be a Broker which will connect Producer to
> Consumer groups.
>
> Am I getting it right ?
>
> Thanks. Any advice would be appreciated.
>
> regards
>
> Chandra R
>

Re: kafka 1:N message delivery pattern

[Joel Koshy <jj...@gmail.com>]

Semantically, yes that's right although it may be more accurate to
think of it as the producers depositing messages at the broker that
can then be consumed by multiple independent consumer groups. The
achieved pattern in your example depends on the view-point. E.g., 1:N
from each producer's point of view, M:N from the broker's point of
view and N:1 from the consumer's point of view.

Joel

On Fri, Sep 9, 2011 at 1:40 PM, Chandra R <my...@gmail.com> wrote:
> Hello All,
>
> I am trying to understand the Kafka 1:N pattern.
> I am trying to understand it in terms of a twitter like app.
>
> For every user there will be 1 Producer. Each follower will be in its own
> consumer group and there will be a Broker which will connect Producer to
> Consumer groups.
>
> Am I getting it right ?
>
> Thanks. Any advice would be appreciated.
>
> regards
>
> Chandra R
>

Re: kafka 1:N message delivery pattern

[Jay Kreps <ja...@gmail.com>]

Each consumer group is a set of one or more machines. The complete stream is
load balanced over the machines in a group (e.g. for a group of N machines,
each machine gets 1/N of the messages). Each group gets a complete copy of
the stream.

So if there is a single consumer group for all consumers this is similar to
a traditional queue.

If each consumer process has its own group id, then this is similar to
traditional pub/sub.

Any case in between allows you to provide a sort of pub/sub that is load
balanced over a cluster of machines.

Does that help?

-Jay

On Fri, Sep 9, 2011 at 1:40 PM, Chandra R <my...@gmail.com> wrote:

> Hello All,
>
> I am trying to understand the Kafka 1:N pattern.
> I am trying to understand it in terms of a twitter like app.
>
> For every user there will be 1 Producer. Each follower will be in its own
> consumer group and there will be a Broker which will connect Producer to
> Consumer groups.
>
> Am I getting it right ?
>
> Thanks. Any advice would be appreciated.
>
> regards
>
> Chandra R
>