You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vahid S Hashemian <va...@us.ibm.com> on 2016/07/20 00:55:31 UTC
[DISCUSS] KIP-70: Revise Partition Assignment Semantics on New Consumer's
Subscription Change
Hi all,
We have started a new KIP under
https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
Your feedback is much appreciated.
Regards,
Vahid Hashemian
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Ismael Juma <is...@juma.me.uk>.
Sounds good Vahid, thanks for doing this. :)
Ismael
On Wed, Jul 27, 2016 at 6:29 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Ismael,
>
> I sent a message to user mailing lists of Spark and Storm a couple of days
> ago and have received one response so far.
> Thread on Spark mailing list:
> https://www.mail-archive.com/user@spark.apache.org/msg54309.html
> Thread on Storm mailing list:
> https://www.mail-archive.com/user@storm.apache.org/msg06850.html
>
> Regards,
> --Vahid
>
>
>
> From: Ismael Juma <is...@juma.me.uk>
> To: dev@kafka.apache.org
> Date: 07/22/2016 01:44 AM
> Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
> Semantics on New Consumer's Subscription Change
> Sent by: ismaelj@gmail.com
>
>
>
> Thanks for the KIP Vahid. The change makes sense. On the compatibility
> front, could we check some of the advanced Kafka users like Storm and
> Spark
> in order to verify if they would be affected?
>
> Ismael
>
> On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi all,
> >
> > We have started a new KIP under
> >
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
> >
> > Your feedback is much appreciated.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Ismael,
I sent a message to user mailing lists of Spark and Storm a couple of days
ago and have received one response so far.
Thread on Spark mailing list:
https://www.mail-archive.com/user@spark.apache.org/msg54309.html
Thread on Storm mailing list:
https://www.mail-archive.com/user@storm.apache.org/msg06850.html
Regards,
--Vahid
From: Ismael Juma <is...@juma.me.uk>
To: dev@kafka.apache.org
Date: 07/22/2016 01:44 AM
Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
Semantics on New Consumer's Subscription Change
Sent by: ismaelj@gmail.com
Thanks for the KIP Vahid. The change makes sense. On the compatibility
front, could we check some of the advanced Kafka users like Storm and
Spark
in order to verify if they would be affected?
Ismael
On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> We have started a new KIP under
>
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
> Your feedback is much appreciated.
>
> Regards,
> Vahid Hashemian
>
>
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Guozhang Wang <wa...@gmail.com>.
This is a nice KIP write-up. LGTM too.
Guozhang
On Fri, Jul 22, 2016 at 12:58 PM, Dana Powers <da...@gmail.com> wrote:
> This is a nice change. Great KIP write up.
>
> -Dana
>
> On Fri, Jul 22, 2016 at 10:07 AM, Vahid S Hashemian
> <va...@us.ibm.com> wrote:
> > Thanks Ismael.
> >
> > What do you think is the best way to check with Storm / Spark users?
> Their
> > mailing list?
> >
> > Thanks.
> >
> > Regards,
> > --Vahid
> >
> >
> >
> >
> > From: Ismael Juma <is...@juma.me.uk>
> > To: dev@kafka.apache.org
> > Date: 07/22/2016 01:44 AM
> > Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
> > Semantics on New Consumer's Subscription Change
> > Sent by: ismaelj@gmail.com
> >
> >
> >
> > Thanks for the KIP Vahid. The change makes sense. On the compatibility
> > front, could we check some of the advanced Kafka users like Storm and
> > Spark
> > in order to verify if they would be affected?
> >
> > Ismael
> >
> > On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
> > vahidhashemian@us.ibm.com> wrote:
> >
> >> Hi all,
> >>
> >> We have started a new KIP under
> >>
> >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
> >
> >>
> >> Your feedback is much appreciated.
> >>
> >> Regards,
> >> Vahid Hashemian
> >>
> >>
> >
> >
> >
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Dana Powers <da...@gmail.com>.
This is a nice change. Great KIP write up.
-Dana
On Fri, Jul 22, 2016 at 10:07 AM, Vahid S Hashemian
<va...@us.ibm.com> wrote:
> Thanks Ismael.
>
> What do you think is the best way to check with Storm / Spark users? Their
> mailing list?
>
> Thanks.
>
> Regards,
> --Vahid
>
>
>
>
> From: Ismael Juma <is...@juma.me.uk>
> To: dev@kafka.apache.org
> Date: 07/22/2016 01:44 AM
> Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
> Semantics on New Consumer's Subscription Change
> Sent by: ismaelj@gmail.com
>
>
>
> Thanks for the KIP Vahid. The change makes sense. On the compatibility
> front, could we check some of the advanced Kafka users like Storm and
> Spark
> in order to verify if they would be affected?
>
> Ismael
>
> On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
>> Hi all,
>>
>> We have started a new KIP under
>>
>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
>>
>> Your feedback is much appreciated.
>>
>> Regards,
>> Vahid Hashemian
>>
>>
>
>
>
>
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Thanks Ismael.
What do you think is the best way to check with Storm / Spark users? Their
mailing list?
Thanks.
Regards,
--Vahid
From: Ismael Juma <is...@juma.me.uk>
To: dev@kafka.apache.org
Date: 07/22/2016 01:44 AM
Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
Semantics on New Consumer's Subscription Change
Sent by: ismaelj@gmail.com
Thanks for the KIP Vahid. The change makes sense. On the compatibility
front, could we check some of the advanced Kafka users like Storm and
Spark
in order to verify if they would be affected?
Ismael
On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> We have started a new KIP under
>
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
> Your feedback is much appreciated.
>
> Regards,
> Vahid Hashemian
>
>
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the KIP Vahid. The change makes sense. On the compatibility
front, could we check some of the advanced Kafka users like Storm and Spark
in order to verify if they would be affected?
Ismael
On Wed, Jul 20, 2016 at 1:55 AM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> We have started a new KIP under
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
> Your feedback is much appreciated.
>
> Regards,
> Vahid Hashemian
>
>
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Thanks Jason / Ewen for your feedback.
I agree that this is more like a bug than anything else and should have
little impact on the users.
Regards,
--Vahid
From: Ewen Cheslack-Postava <ew...@confluent.io>
To: dev@kafka.apache.org
Date: 07/21/2016 10:59 PM
Subject: Re: [DISCUSS] KIP-70: Revise Partition Assignment
Semantics on New Consumer's Subscription Change
Agreed w/ Jason re: compatibility. It seems like such an edge case to
actually rely on this and I'd consider the current behavior essentially a
bug given how surprising it is. While normally a stickler for
compatibility, I think this is a case where its fine to make the change.
-Ewen
On Wed, Jul 20, 2016 at 9:48 AM, Jason Gustafson <ja...@confluent.io>
wrote:
> Hey Vahid,
>
> Thanks for writing this up. This seems like a nice improvement over the
> existing somewhat surprising behavior. Currently if you have a consumer
> which changes subscriptions, then you will need to handle separately any
> cleanup for assigned partitions for topics which are no longer
subscribed.
> With this change, the user can handle this exclusively in the
> onPartitionsRevoked() callback which seems less error prone. This also
> makes it unnecessary for us to do any special handling when autocommit
is
> enabled since all partitions will still be assigned when we do the final
> offset commit prior to rebalancing. The main question mark in my mind is
> compatibility, but it seems unlikely that anyone depends on the current
> behavior. My hunch is that users probably expect it already works this
way,
> so from that perspective, it's almost more of a bug fix.
>
> Thanks,
> Jason
>
> On Tue, Jul 19, 2016 at 5:55 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi all,
> >
> > We have started a new KIP under
> >
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
> >
> > Your feedback is much appreciated.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
>
--
Thanks,
Ewen
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Agreed w/ Jason re: compatibility. It seems like such an edge case to
actually rely on this and I'd consider the current behavior essentially a
bug given how surprising it is. While normally a stickler for
compatibility, I think this is a case where its fine to make the change.
-Ewen
On Wed, Jul 20, 2016 at 9:48 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Vahid,
>
> Thanks for writing this up. This seems like a nice improvement over the
> existing somewhat surprising behavior. Currently if you have a consumer
> which changes subscriptions, then you will need to handle separately any
> cleanup for assigned partitions for topics which are no longer subscribed.
> With this change, the user can handle this exclusively in the
> onPartitionsRevoked() callback which seems less error prone. This also
> makes it unnecessary for us to do any special handling when autocommit is
> enabled since all partitions will still be assigned when we do the final
> offset commit prior to rebalancing. The main question mark in my mind is
> compatibility, but it seems unlikely that anyone depends on the current
> behavior. My hunch is that users probably expect it already works this way,
> so from that perspective, it's almost more of a bug fix.
>
> Thanks,
> Jason
>
> On Tue, Jul 19, 2016 at 5:55 PM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>
> > Hi all,
> >
> > We have started a new KIP under
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
> >
> > Your feedback is much appreciated.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
>
--
Thanks,
Ewen
Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New
Consumer's Subscription Change
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Vahid,
Thanks for writing this up. This seems like a nice improvement over the
existing somewhat surprising behavior. Currently if you have a consumer
which changes subscriptions, then you will need to handle separately any
cleanup for assigned partitions for topics which are no longer subscribed.
With this change, the user can handle this exclusively in the
onPartitionsRevoked() callback which seems less error prone. This also
makes it unnecessary for us to do any special handling when autocommit is
enabled since all partitions will still be assigned when we do the final
offset commit prior to rebalancing. The main question mark in my mind is
compatibility, but it seems unlikely that anyone depends on the current
behavior. My hunch is that users probably expect it already works this way,
so from that perspective, it's almost more of a bug fix.
Thanks,
Jason
On Tue, Jul 19, 2016 at 5:55 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:
> Hi all,
>
> We have started a new KIP under
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
>
> Your feedback is much appreciated.
>
> Regards,
> Vahid Hashemian
>
>