Consumer groups

[Mark <st...@gmail.com>]

How do consumer groups work? Say I have 3 consumers each reading from 
topic1 how do you ensue each consumer is not consuming a message that 
has been consumed by another consumer?

Re: Consumer groups

[Mark <st...@gmail.com>]

I think I understand. It is not possible for two consumers in the same 
consumer group to consume from the same partition therefore duplication 
is not an issue.

Once again my confusion stems from the fact that I've been mostly 
looking into the Ruby consumer/producers. It seems with the Ruby client 
consumer groups are not available and hence only the pub-sub use case is 
available. There also appears to be some other missing functionality 
which further adds to my overall confusion.



On 11/11/11 7:17 PM, Jay Kreps wrote:
> Inder is correct. To elaborate:
>
> Each message is delivered to one consumer in each consumer group.
> Traditional pub/sub delivers each message to each consumer. Traditional
> queuing delivers each message to exactly one consumer. This model
> generalizes both of those and anything in between--if you assign all
> consumers the same consumer group it acts like a queue, if you assign them
> all different consumer groups it acts like pub/sub.
>
> In my experience you rarely want either traditional queues or traditional
> pub/sub in internet-style systems. Each consumer usually MUST load balance
> the consumption over multiple machines for throughput and failover but you
> often do have multiple of these consuming clusters. This is what we are
> attempting to capture.
>
> Let me know if you think we could explain this better, we have really
> struggled to make this clear to people I think.
>
> -Jay
>
> On Fri, Nov 11, 2011 at 6:30 PM, Mark<st...@gmail.com>  wrote:
>
>> How do consumer groups work? Say I have 3 consumers each reading from
>> topic1 how do you ensue each consumer is not consuming a message that has
>> been consumed by another consumer?
>>

Re: Consumer groups

[Jay Kreps <ja...@gmail.com>]

Inder is correct. To elaborate:

Each message is delivered to one consumer in each consumer group.
Traditional pub/sub delivers each message to each consumer. Traditional
queuing delivers each message to exactly one consumer. This model
generalizes both of those and anything in between--if you assign all
consumers the same consumer group it acts like a queue, if you assign them
all different consumer groups it acts like pub/sub.

In my experience you rarely want either traditional queues or traditional
pub/sub in internet-style systems. Each consumer usually MUST load balance
the consumption over multiple machines for throughput and failover but you
often do have multiple of these consuming clusters. This is what we are
attempting to capture.

Let me know if you think we could explain this better, we have really
struggled to make this clear to people I think.

-Jay

On Fri, Nov 11, 2011 at 6:30 PM, Mark <st...@gmail.com> wrote:

> How do consumer groups work? Say I have 3 consumers each reading from
> topic1 how do you ensue each consumer is not consuming a message that has
> been consumed by another consumer?
>