You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mark <st...@gmail.com> on 2013/10/09 19:07:18 UTC

Consumer group questions

This is in regards to consumer group consumption in 0.7.2. 

Say we have 3 machines with 3 partitions in each topic totaling 9 partitions. Now if I create a consumer group with 9 threads on the same machine then all partitions will be read from. Now what happens if I start another 9 threads on a separate machine using the same consume group id? Will the topics be distributed across both machines or will it still be all read from the first process that spawned up the 9 threads?

In general, is it better to have 1 machine running 9 threads to read all partitions or 9 machines running 1 thread?

Is there a wiki for 0.7.2? I've been trying to work with (https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example) and it appears to be for 0.8

Thanks

Re: Consumer group questions

Posted by Neha Narkhede <ne...@gmail.com>.
If one machine crashes is there anything I can do to make sure those 3
partitions are still read from?

If all those consumers are part of the same group, nothing. It will
automatically rebalance to consume the partitions that were consumed by the
failed client.

Thanks,
Neha


On Wed, Oct 9, 2013 at 4:02 PM, Mark <st...@gmail.com> wrote:

> > Will the topics be distributed across both machines or will it still be
> all
> > read from the first process that spawned up the 9 threads?
> >
> > It will read from the first process that spawned 9 threads.
>
> I should have actually asked the opposite of this. Say I have a consumer
> running 3 threads on 3 different machines. If one machine crashes is there
> anything I can do to make sure those 3 partitions are still read from?
>
> On Oct 9, 2013, at 1:18 PM, Neha Narkhede <ne...@gmail.com> wrote:
>
> > Will the topics be distributed across both machines or will it still be
> all
> > read from the first process that spawned up the 9 threads?
> >
> > It will read from the first process that spawned 9 threads.
> >
> > In general, is it better to have 1 machine running 9 threads to read all
> > partitions or 9 machines running 1 thread?
> >
> > It really depends on the network bandwidth and CPU usage of your
> > application.
> >
> > I've been trying to work with (
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> )
> > and it appears to be for 0.8
> >
> > The code is not different from the high level consumer in 0.7. You can
> see
> > the 0.7 configs here - http://kafka.apache.org/07/configuration.html
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Oct 9, 2013 at 10:07 AM, Mark <st...@gmail.com> wrote:
> >
> >> This is in regards to consumer group consumption in 0.7.2.
> >>
> >> Say we have 3 machines with 3 partitions in each topic totaling 9
> >> partitions. Now if I create a consumer group with 9 threads on the same
> >> machine then all partitions will be read from. Now what happens if I
> start
> >> another 9 threads on a separate machine using the same consume group id?
> >> Will the topics be distributed across both machines or will it still be
> all
> >> read from the first process that spawned up the 9 threads?
> >>
> >> In general, is it better to have 1 machine running 9 threads to read all
> >> partitions or 9 machines running 1 thread?
> >>
> >> Is there a wiki for 0.7.2? I've been trying to work with (
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
> >> and it appears to be for 0.8
> >>
> >> Thanks
>
>

Re: Consumer group questions

Posted by Mark <st...@gmail.com>.
> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
> 
> It will read from the first process that spawned 9 threads.

I should have actually asked the opposite of this. Say I have a consumer running 3 threads on 3 different machines. If one machine crashes is there anything I can do to make sure those 3 partitions are still read from?

On Oct 9, 2013, at 1:18 PM, Neha Narkhede <ne...@gmail.com> wrote:

> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
> 
> It will read from the first process that spawned 9 threads.
> 
> In general, is it better to have 1 machine running 9 threads to read all
> partitions or 9 machines running 1 thread?
> 
> It really depends on the network bandwidth and CPU usage of your
> application.
> 
> I've been trying to work with (
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
> and it appears to be for 0.8
> 
> The code is not different from the high level consumer in 0.7. You can see
> the 0.7 configs here - http://kafka.apache.org/07/configuration.html
> 
> Thanks,
> Neha
> 
> 
> On Wed, Oct 9, 2013 at 10:07 AM, Mark <st...@gmail.com> wrote:
> 
>> This is in regards to consumer group consumption in 0.7.2.
>> 
>> Say we have 3 machines with 3 partitions in each topic totaling 9
>> partitions. Now if I create a consumer group with 9 threads on the same
>> machine then all partitions will be read from. Now what happens if I start
>> another 9 threads on a separate machine using the same consume group id?
>> Will the topics be distributed across both machines or will it still be all
>> read from the first process that spawned up the 9 threads?
>> 
>> In general, is it better to have 1 machine running 9 threads to read all
>> partitions or 9 machines running 1 thread?
>> 
>> Is there a wiki for 0.7.2? I've been trying to work with (
>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
>> and it appears to be for 0.8
>> 
>> Thanks

Re: Consumer group questions

Posted by Neha Narkhede <ne...@gmail.com>.
Will the topics be distributed across both machines or will it still be all
read from the first process that spawned up the 9 threads?

It will read from the first process that spawned 9 threads.

In general, is it better to have 1 machine running 9 threads to read all
partitions or 9 machines running 1 thread?

It really depends on the network bandwidth and CPU usage of your
application.

I've been trying to work with (
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
and it appears to be for 0.8

The code is not different from the high level consumer in 0.7. You can see
the 0.7 configs here - http://kafka.apache.org/07/configuration.html

Thanks,
Neha


On Wed, Oct 9, 2013 at 10:07 AM, Mark <st...@gmail.com> wrote:

> This is in regards to consumer group consumption in 0.7.2.
>
> Say we have 3 machines with 3 partitions in each topic totaling 9
> partitions. Now if I create a consumer group with 9 threads on the same
> machine then all partitions will be read from. Now what happens if I start
> another 9 threads on a separate machine using the same consume group id?
> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
>
> In general, is it better to have 1 machine running 9 threads to read all
> partitions or 9 machines running 1 thread?
>
> Is there a wiki for 0.7.2? I've been trying to work with (
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
> and it appears to be for 0.8
>
> Thanks