You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Withers, Robert" <Ro...@dish.com> on 2013/05/16 15:09:32 UTC
only-once consumer groups
is it technically feasible to use an only-once simple consumer within a consumer group?
thanks,
rob
RE: only-once consumer groups
Posted by Rob Withers <re...@gmail.com>.
Right, Neha, consumer groups are from 7, while replicas are in 8. Does this
mean the simple consumer in 8 can recognize a leader change?
Thanks,
rob
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Monday, May 20, 2013 8:00 PM
> To: users@kafka.apache.org
> Subject: RE: only-once consumer groups
>
> Rob,
>
> A consumer rebalances whenever a consumer process dies or a new
> consumer process joins the group. The details of the algorithm can be
found
> here http://kafka.apache.org/07/design.html
>
> Thanks,
> Neha
> On May 20, 2013 6:45 PM, "Rob Withers" <re...@gmail.com> wrote:
>
> > That page is packed full of super design! Many of these features we
> > would find useful, I think. One thing I found myself not knowing is
> > what a consumer rebalance actually is.
> >
> > Is a rebalance when the thread that is consuming a particular
> > partition dies, this is detected and the partition is reassigned to a
> > new thread, thus violating manual partitioning?
> >
> > Or is a rebalance when a broker dies and new leader partitions are
elected?
> > The simple consumer must get told of a leader election and this is
> > nothing to do with rebalance, is it so?
> >
> > Thanks,
> > rob
> >
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Friday, May 17, 2013 7:32 AM
> > > To: users@kafka.apache.org
> > > Subject: RE: only-once consumer groups
> > >
> > > We spent some time thinking about consolidating the high level and
> > > low
> > level
> > > consumer APIs. It will be great if you can read the wiki and provide
> > feedback
> > > -
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re
> > > -
> > > Design
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com> wrote:
> > >
> > > > We want to ensure only-once message processing, but we also want
> > > > the benefit of rebalancing. commitOffsets updates all partitions
> > > > from out of a connector instance. We want to commit the offset
> > > > for just the partition that delivered a message to the iterator,
> > > > even if several fetchers are feeding a thread. Perhaps the
> > > > message metadata contains the partition a msg came from; could you
> > > > not just update the offset for that partition if the property
> > > > "only.once=true" is sent to the consumer connector?
> > > >
> > > > Thanks,
> > > > rob
> > > >
> > > > > -----Original Message-----
> > > > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: only-once consumer groups
> > > > >
> > > > > Can you describe your requirements in a little more detail?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > > <Ro...@dish.com>
> > > > > wrote:
> > > > >
> > > > > > is it technically feasible to use an only-once simple consumer
> > > > > > within a consumer group?
> > > > > >
> > > > > > thanks,
> > > > > > rob
> > > >
> > > >
> >
> >
RE: only-once consumer groups
Posted by Rob Withers <re...@gmail.com>.
Sorry for being unclear Neha. I meant that I had forgotten that the
introduction of replicas is happening in 0.8 and I was confusing the two.
Thanks,
rob
> -----Original Message-----
> From: Rob Withers [mailto:reefedjib@gmail.com]
> Sent: Monday, May 20, 2013 8:21 PM
> To: 'users@kafka.apache.org'
> Subject: RE: only-once consumer groups
>
> Right, Neha, consumer groups are from 7, while replicas are in 8. Does
this
> mean the simple consumer in 8 can recognize a leader change?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Monday, May 20, 2013 8:00 PM
> > To: users@kafka.apache.org
> > Subject: RE: only-once consumer groups
> >
> > Rob,
> >
> > A consumer rebalances whenever a consumer process dies or a new
> > consumer process joins the group. The details of the algorithm can be
> > found here http://kafka.apache.org/07/design.html
> >
> > Thanks,
> > Neha
> > On May 20, 2013 6:45 PM, "Rob Withers" <re...@gmail.com> wrote:
> >
> > > That page is packed full of super design! Many of these features we
> > > would find useful, I think. One thing I found myself not knowing is
> > > what a consumer rebalance actually is.
> > >
> > > Is a rebalance when the thread that is consuming a particular
> > > partition dies, this is detected and the partition is reassigned to
> > > a new thread, thus violating manual partitioning?
> > >
> > > Or is a rebalance when a broker dies and new leader partitions are
> elected?
> > > The simple consumer must get told of a leader election and this is
> > > nothing to do with rebalance, is it so?
> > >
> > > Thanks,
> > > rob
> > >
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > > Sent: Friday, May 17, 2013 7:32 AM
> > > > To: users@kafka.apache.org
> > > > Subject: RE: only-once consumer groups
> > > >
> > > > We spent some time thinking about consolidating the high level and
> > > > low
> > > level
> > > > consumer APIs. It will be great if you can read the wiki and
> > > > provide
> > > feedback
> > > > -
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re
> > > > -
> > > > Design
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com>
> wrote:
> > > >
> > > > > We want to ensure only-once message processing, but we also want
> > > > > the benefit of rebalancing. commitOffsets updates all
> > > > > partitions from out of a connector instance. We want to commit
> > > > > the offset for just the partition that delivered a message to
> > > > > the iterator, even if several fetchers are feeding a thread.
> > > > > Perhaps the message metadata contains the partition a msg came
> > > > > from; could you not just update the offset for that partition if
> > > > > the property "only.once=true" is sent to the consumer connector?
> > > > >
> > > > > Thanks,
> > > > > rob
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: only-once consumer groups
> > > > > >
> > > > > > Can you describe your requirements in a little more detail?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > > > <Ro...@dish.com>
> > > > > > wrote:
> > > > > >
> > > > > > > is it technically feasible to use an only-once simple
> > > > > > > consumer within a consumer group?
> > > > > > >
> > > > > > > thanks,
> > > > > > > rob
> > > > >
> > > > >
> > >
> > >
RE: only-once consumer groups
Posted by Neha Narkhede <ne...@gmail.com>.
Rob,
A consumer rebalances whenever a consumer process dies or a new consumer
process joins the group. The details of the algorithm can be found here
http://kafka.apache.org/07/design.html
Thanks,
Neha
On May 20, 2013 6:45 PM, "Rob Withers" <re...@gmail.com> wrote:
> That page is packed full of super design! Many of these features we would
> find useful, I think. One thing I found myself not knowing is what a
> consumer rebalance actually is.
>
> Is a rebalance when the thread that is consuming a particular partition
> dies, this is detected and the partition is reassigned to a new thread,
> thus
> violating manual partitioning?
>
> Or is a rebalance when a broker dies and new leader partitions are elected?
> The simple consumer must get told of a leader election and this is nothing
> to do with rebalance, is it so?
>
> Thanks,
> rob
>
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Friday, May 17, 2013 7:32 AM
> > To: users@kafka.apache.org
> > Subject: RE: only-once consumer groups
> >
> > We spent some time thinking about consolidating the high level and low
> level
> > consumer APIs. It will be great if you can read the wiki and provide
> feedback
> > - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-
> > Design
> >
> > Thanks,
> > Neha
> > On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com> wrote:
> >
> > > We want to ensure only-once message processing, but we also want the
> > > benefit of rebalancing. commitOffsets updates all partitions from out
> > > of a connector instance. We want to commit the offset for just the
> > > partition that delivered a message to the iterator, even if several
> > > fetchers are feeding a thread. Perhaps the message metadata contains
> > > the partition a msg came from; could you not just update the offset
> > > for that partition if the property "only.once=true" is sent to the
> > > consumer connector?
> > >
> > > Thanks,
> > > rob
> > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > > Sent: Thursday, May 16, 2013 10:03 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: only-once consumer groups
> > > >
> > > > Can you describe your requirements in a little more detail?
> > > >
> > > > Thanks,
> > > > Neha
> > > > On May 16, 2013 6:11 AM, "Withers, Robert"
> > <Ro...@dish.com>
> > > > wrote:
> > > >
> > > > > is it technically feasible to use an only-once simple consumer
> > > > > within a consumer group?
> > > > >
> > > > > thanks,
> > > > > rob
> > >
> > >
>
>
RE: only-once consumer groups
Posted by Rob Withers <re...@gmail.com>.
That page is packed full of super design! Many of these features we would
find useful, I think. One thing I found myself not knowing is what a
consumer rebalance actually is.
Is a rebalance when the thread that is consuming a particular partition
dies, this is detected and the partition is reassigned to a new thread, thus
violating manual partitioning?
Or is a rebalance when a broker dies and new leader partitions are elected?
The simple consumer must get told of a leader election and this is nothing
to do with rebalance, is it so?
Thanks,
rob
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, May 17, 2013 7:32 AM
> To: users@kafka.apache.org
> Subject: RE: only-once consumer groups
>
> We spent some time thinking about consolidating the high level and low
level
> consumer APIs. It will be great if you can read the wiki and provide
feedback
> - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-
> Design
>
> Thanks,
> Neha
> On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com> wrote:
>
> > We want to ensure only-once message processing, but we also want the
> > benefit of rebalancing. commitOffsets updates all partitions from out
> > of a connector instance. We want to commit the offset for just the
> > partition that delivered a message to the iterator, even if several
> > fetchers are feeding a thread. Perhaps the message metadata contains
> > the partition a msg came from; could you not just update the offset
> > for that partition if the property "only.once=true" is sent to the
> > consumer connector?
> >
> > Thanks,
> > rob
> >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Thursday, May 16, 2013 10:03 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: only-once consumer groups
> > >
> > > Can you describe your requirements in a little more detail?
> > >
> > > Thanks,
> > > Neha
> > > On May 16, 2013 6:11 AM, "Withers, Robert"
> <Ro...@dish.com>
> > > wrote:
> > >
> > > > is it technically feasible to use an only-once simple consumer
> > > > within a consumer group?
> > > >
> > > > thanks,
> > > > rob
> >
> >
RE: only-once consumer groups
Posted by "Withers, Robert" <Ro...@dish.com>.
Absolutely, I will. However, I am pretty full today and I'll need more than 15 minutes to absorb what you have written, which looks very comprehensive.
thanks,
rob
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Friday, May 17, 2013 7:32 AM
To: users@kafka.apache.org
Subject: RE: only-once consumer groups
We spent some time thinking about consolidating the high level and low
level consumer APIs. It will be great if you can read the wiki and provide
feedback -
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
Thanks,
Neha
On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com> wrote:
> We want to ensure only-once message processing, but we also want the
> benefit
> of rebalancing. commitOffsets updates all partitions from out of a
> connector instance. We want to commit the offset for just the partition
> that delivered a message to the iterator, even if several fetchers are
> feeding a thread. Perhaps the message metadata contains the partition a
> msg
> came from; could you not just update the offset for that partition if the
> property "only.once=true" is sent to the consumer connector?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Thursday, May 16, 2013 10:03 PM
> > To: users@kafka.apache.org
> > Subject: Re: only-once consumer groups
> >
> > Can you describe your requirements in a little more detail?
> >
> > Thanks,
> > Neha
> > On May 16, 2013 6:11 AM, "Withers, Robert" <Ro...@dish.com>
> > wrote:
> >
> > > is it technically feasible to use an only-once simple consumer within
> > > a consumer group?
> > >
> > > thanks,
> > > rob
>
>
RE: only-once consumer groups
Posted by Neha Narkhede <ne...@gmail.com>.
We spent some time thinking about consolidating the high level and low
level consumer APIs. It will be great if you can read the wiki and provide
feedback -
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
Thanks,
Neha
On May 16, 2013 10:29 PM, "Rob Withers" <re...@gmail.com> wrote:
> We want to ensure only-once message processing, but we also want the
> benefit
> of rebalancing. commitOffsets updates all partitions from out of a
> connector instance. We want to commit the offset for just the partition
> that delivered a message to the iterator, even if several fetchers are
> feeding a thread. Perhaps the message metadata contains the partition a
> msg
> came from; could you not just update the offset for that partition if the
> property "only.once=true" is sent to the consumer connector?
>
> Thanks,
> rob
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Thursday, May 16, 2013 10:03 PM
> > To: users@kafka.apache.org
> > Subject: Re: only-once consumer groups
> >
> > Can you describe your requirements in a little more detail?
> >
> > Thanks,
> > Neha
> > On May 16, 2013 6:11 AM, "Withers, Robert" <Ro...@dish.com>
> > wrote:
> >
> > > is it technically feasible to use an only-once simple consumer within
> > > a consumer group?
> > >
> > > thanks,
> > > rob
>
>
RE: only-once consumer groups
Posted by Rob Withers <re...@gmail.com>.
We want to ensure only-once message processing, but we also want the benefit
of rebalancing. commitOffsets updates all partitions from out of a
connector instance. We want to commit the offset for just the partition
that delivered a message to the iterator, even if several fetchers are
feeding a thread. Perhaps the message metadata contains the partition a msg
came from; could you not just update the offset for that partition if the
property "only.once=true" is sent to the consumer connector?
Thanks,
rob
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Thursday, May 16, 2013 10:03 PM
> To: users@kafka.apache.org
> Subject: Re: only-once consumer groups
>
> Can you describe your requirements in a little more detail?
>
> Thanks,
> Neha
> On May 16, 2013 6:11 AM, "Withers, Robert" <Ro...@dish.com>
> wrote:
>
> > is it technically feasible to use an only-once simple consumer within
> > a consumer group?
> >
> > thanks,
> > rob
Re: only-once consumer groups
Posted by Neha Narkhede <ne...@gmail.com>.
Can you describe your requirements in a little more detail?
Thanks,
Neha
On May 16, 2013 6:11 AM, "Withers, Robert" <Ro...@dish.com> wrote:
> is it technically feasible to use an only-once simple consumer within a
> consumer group?
>
> thanks,
> rob