You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sam William <sa...@stumbleupon.com> on 2012/06/12 04:11:26 UTC
High level consumer Question
Hi,
Im trying out the high level consumer using the ConsumerConnector API . I have a topic with 100 partitions . What does it mean if I create messagestreams with
consumerConnector.createMessageStreams(Predef.Map(topic -> partitions), decoder)
where partitions < 100 (say 50). does the consumer read messages only from 50 of the 100 partitions ?
Sam William
sampd@stumbleupon.com
Re: High level consumer Question
Posted by Jay Kreps <ja...@gmail.com>.
Yes.
-Jay
On Wed, Jun 13, 2012 at 4:35 PM, Sam William <sa...@stumbleupon.com> wrote:
> So lets say I have a topic with 100 partitions. And Im running a
> high-kevel consumer with 25 threads. If I start a new consumer process with
> the same groupid and say this has another 10 threads. I can expect some
> partitions to be reassigned to the new consumers right ?
>
> On Jun 11, 2012, at 8:05 PM, Sam William wrote:
>
> > Thanks Jay.. I had made such an assumption and seems like it was valid.
> >
> > Sam
> > On Jun 11, 2012, at 7:52 PM, Jay Kreps wrote:
> >
> >> Hi Sam,
> >>
> >> We assign partitions to consumer threads so that each partition has a
> >> consumer. If you have more partitions then threads, some threads will
> get
> >> multiple partitions (nothing wrong with this); if you have more threads
> >> than partitions some thread will not get any partitions and will be
> idle.
> >> But never will one partition go to more than one thread (since the
> >> consumption is meant to be ordered within that partition, splitting it
> >> would likely break this).
> >>
> >> -Jay
> >>
> >> On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sa...@stumbleupon.com>
> wrote:
> >>
> >>> Hi,
> >>> Im trying out the high level consumer using the ConsumerConnector API
> .
> >>> I have a topic with 100 partitions . What does it mean if I create
> >>> messagestreams with
> >>>
> >>>
> >>> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions),
> >>> decoder)
> >>>
> >>> where partitions < 100 (say 50). does the consumer read messages only
> >>> from 50 of the 100 partitions ?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Sam William
> >>> sampd@stumbleupon.com
> >>>
> >>>
> >>>
> >>>
> >
> > Sam William
> > sampd@stumbleupon.com
> >
> >
> >
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: High level consumer Question
Posted by Sam William <sa...@stumbleupon.com>.
So lets say I have a topic with 100 partitions. And Im running a high-kevel consumer with 25 threads. If I start a new consumer process with the same groupid and say this has another 10 threads. I can expect some partitions to be reassigned to the new consumers right ?
On Jun 11, 2012, at 8:05 PM, Sam William wrote:
> Thanks Jay.. I had made such an assumption and seems like it was valid.
>
> Sam
> On Jun 11, 2012, at 7:52 PM, Jay Kreps wrote:
>
>> Hi Sam,
>>
>> We assign partitions to consumer threads so that each partition has a
>> consumer. If you have more partitions then threads, some threads will get
>> multiple partitions (nothing wrong with this); if you have more threads
>> than partitions some thread will not get any partitions and will be idle.
>> But never will one partition go to more than one thread (since the
>> consumption is meant to be ordered within that partition, splitting it
>> would likely break this).
>>
>> -Jay
>>
>> On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sa...@stumbleupon.com> wrote:
>>
>>> Hi,
>>> Im trying out the high level consumer using the ConsumerConnector API .
>>> I have a topic with 100 partitions . What does it mean if I create
>>> messagestreams with
>>>
>>>
>>> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions),
>>> decoder)
>>>
>>> where partitions < 100 (say 50). does the consumer read messages only
>>> from 50 of the 100 partitions ?
>>>
>>>
>>>
>>>
>>>
>>> Sam William
>>> sampd@stumbleupon.com
>>>
>>>
>>>
>>>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
Sam William
sampd@stumbleupon.com
Re: High level consumer Question
Posted by Sam William <sa...@stumbleupon.com>.
Thanks Jay.. I had made such an assumption and seems like it was valid.
Sam
On Jun 11, 2012, at 7:52 PM, Jay Kreps wrote:
> Hi Sam,
>
> We assign partitions to consumer threads so that each partition has a
> consumer. If you have more partitions then threads, some threads will get
> multiple partitions (nothing wrong with this); if you have more threads
> than partitions some thread will not get any partitions and will be idle.
> But never will one partition go to more than one thread (since the
> consumption is meant to be ordered within that partition, splitting it
> would likely break this).
>
> -Jay
>
> On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sa...@stumbleupon.com> wrote:
>
>> Hi,
>> Im trying out the high level consumer using the ConsumerConnector API .
>> I have a topic with 100 partitions . What does it mean if I create
>> messagestreams with
>>
>>
>> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions),
>> decoder)
>>
>> where partitions < 100 (say 50). does the consumer read messages only
>> from 50 of the 100 partitions ?
>>
>>
>>
>>
>>
>> Sam William
>> sampd@stumbleupon.com
>>
>>
>>
>>
Sam William
sampd@stumbleupon.com
Re: High level consumer Question
Posted by Jay Kreps <ja...@gmail.com>.
Hi Sam,
We assign partitions to consumer threads so that each partition has a
consumer. If you have more partitions then threads, some threads will get
multiple partitions (nothing wrong with this); if you have more threads
than partitions some thread will not get any partitions and will be idle.
But never will one partition go to more than one thread (since the
consumption is meant to be ordered within that partition, splitting it
would likely break this).
-Jay
On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sa...@stumbleupon.com> wrote:
> Hi,
> Im trying out the high level consumer using the ConsumerConnector API .
> I have a topic with 100 partitions . What does it mean if I create
> messagestreams with
>
>
> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions),
> decoder)
>
> where partitions < 100 (say 50). does the consumer read messages only
> from 50 of the 100 partitions ?
>
>
>
>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>