You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "M. Manna" <ma...@gmail.com> on 2019/04/04 19:24:21 UTC
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Hello,
Trying to revive this thread again. Would anyone be interested in having
this KiP through
Thanks,
On Fri, 25 Jan 2019 at 16:44, M. Manna <ma...@gmail.com> wrote:
> Hello,
>
> I am trying to revive this thread. I only got 1 binding vote so far.
>
> Please feel free to revisit and comment here.
>
> Thanks,
>
> On Thu, 25 Oct 2018 at 00:15, M. Manna <ma...@gmail.com> wrote:
>
>> Hey IJ,
>>
>> Thanks for your interest in the KIP.
>>
>> My point was simply that the round-robin should happen even if the key is
>> not null. As for the importance of key in our case, we treat the key as
>> metadata. Each key is composed of certain info which are parsed by our
>> consumer thread. We will then determine whether it's an actionable message
>> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
>> append this metadata with the record and parse it there?". But that means
>> the following:
>>
>> 1) I'm always passing null key to achieve this - I would like to pass
>> Null/Not-Null/Other key i.e. flexibility
>> 2) Suppose the message size is 99 KB and and max message bytes allowed is
>> 100K. Now prefixing metadata with message results into the actual message
>> being 101K. This will fail at producer level and cause a retry/log this in
>> our DB for future pickup.
>>
>> To avoid all these, we are simply proposing this new partitioner class.
>> but all Kafka new releases will still have DefaultPartitioner as default,
>> unless they change the prop file to use our new class.
>>
>> Regards,
>>
>> On Sun, 21 Oct 2018 at 04:05, Ismael Juma <is...@juma.me.uk> wrote:
>>
>>> Thanks for the KIP. Can you please elaborate on the need for the key in
>>> this case? The KIP simply states that the key is needed for metadata, but
>>> doesn't give any more details.
>>>
>>> Ismael
>>>
>>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna <ma...@gmail.com> wrote:
>>>
>>> > Hello,
>>> >
>>> > I have made necessary changes as per the original discussion thread,
>>> and
>>> > would like to put it for votes.
>>> >
>>> > Thank you very much for your suggestion and guidance so far.
>>> >
>>> > Regards,
>>> >
>>>
>>
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Posted by "M. Manna" <ma...@gmail.com>.
I have now submitted the PR for review. Thanks Matthias for pointing out
that KAFKA-3333 was raised to address the same.
https://github.com/apache/kafka/pull/6771
if someone reviews after Jenkins build is complete, I would appreciate it.
Thanks,
On Fri, 17 May 2019 at 22:18, M. Manna <ma...@gmail.com> wrote:
> Apologies for the delay. As per the original thread, there have been 3
> binding votes.
>
> I will be closing this and update the confluence page with the results.
> Also, I will be submitting the PR soon.
>
> Regards,
>
> On Fri, 5 Apr 2019 at 00:18, M. Manna <ma...@gmail.com> wrote:
>
>> Thanks Harsha.
>>
>> As per your comments, I have counted 3 binding votes so far.
>>
>> Thanks everyone for your comments and support. I’ll update the kip next
>> morning and do the needful.
>>
>> Regards,
>>
>> On Thu, 4 Apr 2019 at 22:10, Harsha <ka...@harsha.io> wrote:
>>
>>> Looks like the KIP is passed with 3 binding votes. From Matthias, Bill
>>> Bejeck and myself you got 3 binding votes.
>>> You can do the full tally of the votes and send out a close of vote
>>> thread.
>>>
>>> Thanks,
>>> Harsha
>>>
>>> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
>>> > Hello,
>>> >
>>> > Trying to revive this thread again. Would anyone be interested in
>>> having
>>> > this KiP through
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > On Fri, 25 Jan 2019 at 16:44, M. Manna <ma...@gmail.com> wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > I am trying to revive this thread. I only got 1 binding vote so far.
>>> > >
>>> > > Please feel free to revisit and comment here.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > On Thu, 25 Oct 2018 at 00:15, M. Manna <ma...@gmail.com> wrote:
>>> > >
>>> > >> Hey IJ,
>>> > >>
>>> > >> Thanks for your interest in the KIP.
>>> > >>
>>> > >> My point was simply that the round-robin should happen even if the
>>> key is
>>> > >> not null. As for the importance of key in our case, we treat the
>>> key as
>>> > >> metadata. Each key is composed of certain info which are parsed by
>>> our
>>> > >> consumer thread. We will then determine whether it's an actionable
>>> message
>>> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why
>>> not
>>> > >> append this metadata with the record and parse it there?". But that
>>> means
>>> > >> the following:
>>> > >>
>>> > >> 1) I'm always passing null key to achieve this - I would like to
>>> pass
>>> > >> Null/Not-Null/Other key i.e. flexibility
>>> > >> 2) Suppose the message size is 99 KB and and max message bytes
>>> allowed is
>>> > >> 100K. Now prefixing metadata with message results into the actual
>>> message
>>> > >> being 101K. This will fail at producer level and cause a retry/log
>>> this in
>>> > >> our DB for future pickup.
>>> > >>
>>> > >> To avoid all these, we are simply proposing this new partitioner
>>> class.
>>> > >> but all Kafka new releases will still have DefaultPartitioner as
>>> default,
>>> > >> unless they change the prop file to use our new class.
>>> > >>
>>> > >> Regards,
>>> > >>
>>> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma <is...@juma.me.uk>
>>> wrote:
>>> > >>
>>> > >>> Thanks for the KIP. Can you please elaborate on the need for the
>>> key in
>>> > >>> this case? The KIP simply states that the key is needed for
>>> metadata, but
>>> > >>> doesn't give any more details.
>>> > >>>
>>> > >>> Ismael
>>> > >>>
>>> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna <ma...@gmail.com>
>>> wrote:
>>> > >>>
>>> > >>> > Hello,
>>> > >>> >
>>> > >>> > I have made necessary changes as per the original discussion
>>> thread,
>>> > >>> and
>>> > >>> > would like to put it for votes.
>>> > >>> >
>>> > >>> > Thank you very much for your suggestion and guidance so far.
>>> > >>> >
>>> > >>> > Regards,
>>> > >>> >
>>> > >>>
>>> > >>
>>> >
>>>
>>
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Posted by "M. Manna" <ma...@gmail.com>.
Apologies for the delay. As per the original thread, there have been 3
binding votes.
I will be closing this and update the confluence page with the results.
Also, I will be submitting the PR soon.
Regards,
On Fri, 5 Apr 2019 at 00:18, M. Manna <ma...@gmail.com> wrote:
> Thanks Harsha.
>
> As per your comments, I have counted 3 binding votes so far.
>
> Thanks everyone for your comments and support. I’ll update the kip next
> morning and do the needful.
>
> Regards,
>
> On Thu, 4 Apr 2019 at 22:10, Harsha <ka...@harsha.io> wrote:
>
>> Looks like the KIP is passed with 3 binding votes. From Matthias, Bill
>> Bejeck and myself you got 3 binding votes.
>> You can do the full tally of the votes and send out a close of vote
>> thread.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
>> > Hello,
>> >
>> > Trying to revive this thread again. Would anyone be interested in having
>> > this KiP through
>> >
>> >
>> > Thanks,
>> >
>> > On Fri, 25 Jan 2019 at 16:44, M. Manna <ma...@gmail.com> wrote:
>> >
>> > > Hello,
>> > >
>> > > I am trying to revive this thread. I only got 1 binding vote so far.
>> > >
>> > > Please feel free to revisit and comment here.
>> > >
>> > > Thanks,
>> > >
>> > > On Thu, 25 Oct 2018 at 00:15, M. Manna <ma...@gmail.com> wrote:
>> > >
>> > >> Hey IJ,
>> > >>
>> > >> Thanks for your interest in the KIP.
>> > >>
>> > >> My point was simply that the round-robin should happen even if the
>> key is
>> > >> not null. As for the importance of key in our case, we treat the key
>> as
>> > >> metadata. Each key is composed of certain info which are parsed by
>> our
>> > >> consumer thread. We will then determine whether it's an actionable
>> message
>> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why
>> not
>> > >> append this metadata with the record and parse it there?". But that
>> means
>> > >> the following:
>> > >>
>> > >> 1) I'm always passing null key to achieve this - I would like to pass
>> > >> Null/Not-Null/Other key i.e. flexibility
>> > >> 2) Suppose the message size is 99 KB and and max message bytes
>> allowed is
>> > >> 100K. Now prefixing metadata with message results into the actual
>> message
>> > >> being 101K. This will fail at producer level and cause a retry/log
>> this in
>> > >> our DB for future pickup.
>> > >>
>> > >> To avoid all these, we are simply proposing this new partitioner
>> class.
>> > >> but all Kafka new releases will still have DefaultPartitioner as
>> default,
>> > >> unless they change the prop file to use our new class.
>> > >>
>> > >> Regards,
>> > >>
>> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma <is...@juma.me.uk> wrote:
>> > >>
>> > >>> Thanks for the KIP. Can you please elaborate on the need for the
>> key in
>> > >>> this case? The KIP simply states that the key is needed for
>> metadata, but
>> > >>> doesn't give any more details.
>> > >>>
>> > >>> Ismael
>> > >>>
>> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna <ma...@gmail.com> wrote:
>> > >>>
>> > >>> > Hello,
>> > >>> >
>> > >>> > I have made necessary changes as per the original discussion
>> thread,
>> > >>> and
>> > >>> > would like to put it for votes.
>> > >>> >
>> > >>> > Thank you very much for your suggestion and guidance so far.
>> > >>> >
>> > >>> > Regards,
>> > >>> >
>> > >>>
>> > >>
>> >
>>
>
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Posted by "M. Manna" <ma...@gmail.com>.
Thanks Harsha.
As per your comments, I have counted 3 binding votes so far.
Thanks everyone for your comments and support. I’ll update the kip next
morning and do the needful.
Regards,
On Thu, 4 Apr 2019 at 22:10, Harsha <ka...@harsha.io> wrote:
> Looks like the KIP is passed with 3 binding votes. From Matthias, Bill
> Bejeck and myself you got 3 binding votes.
> You can do the full tally of the votes and send out a close of vote thread.
>
> Thanks,
> Harsha
>
> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
> > Hello,
> >
> > Trying to revive this thread again. Would anyone be interested in having
> > this KiP through
> >
> >
> > Thanks,
> >
> > On Fri, 25 Jan 2019 at 16:44, M. Manna <ma...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > I am trying to revive this thread. I only got 1 binding vote so far.
> > >
> > > Please feel free to revisit and comment here.
> > >
> > > Thanks,
> > >
> > > On Thu, 25 Oct 2018 at 00:15, M. Manna <ma...@gmail.com> wrote:
> > >
> > >> Hey IJ,
> > >>
> > >> Thanks for your interest in the KIP.
> > >>
> > >> My point was simply that the round-robin should happen even if the
> key is
> > >> not null. As for the importance of key in our case, we treat the key
> as
> > >> metadata. Each key is composed of certain info which are parsed by our
> > >> consumer thread. We will then determine whether it's an actionable
> message
> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> > >> append this metadata with the record and parse it there?". But that
> means
> > >> the following:
> > >>
> > >> 1) I'm always passing null key to achieve this - I would like to pass
> > >> Null/Not-Null/Other key i.e. flexibility
> > >> 2) Suppose the message size is 99 KB and and max message bytes
> allowed is
> > >> 100K. Now prefixing metadata with message results into the actual
> message
> > >> being 101K. This will fail at producer level and cause a retry/log
> this in
> > >> our DB for future pickup.
> > >>
> > >> To avoid all these, we are simply proposing this new partitioner
> class.
> > >> but all Kafka new releases will still have DefaultPartitioner as
> default,
> > >> unless they change the prop file to use our new class.
> > >>
> > >> Regards,
> > >>
> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma <is...@juma.me.uk> wrote:
> > >>
> > >>> Thanks for the KIP. Can you please elaborate on the need for the key
> in
> > >>> this case? The KIP simply states that the key is needed for
> metadata, but
> > >>> doesn't give any more details.
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna <ma...@gmail.com> wrote:
> > >>>
> > >>> > Hello,
> > >>> >
> > >>> > I have made necessary changes as per the original discussion
> thread,
> > >>> and
> > >>> > would like to put it for votes.
> > >>> >
> > >>> > Thank you very much for your suggestion and guidance so far.
> > >>> >
> > >>> > Regards,
> > >>> >
> > >>>
> > >>
> >
>
Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection
Posted by Harsha <ka...@harsha.io>.
Looks like the KIP is passed with 3 binding votes. From Matthias, Bill Bejeck and myself you got 3 binding votes.
You can do the full tally of the votes and send out a close of vote thread.
Thanks,
Harsha
On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
> Hello,
>
> Trying to revive this thread again. Would anyone be interested in having
> this KiP through
>
>
> Thanks,
>
> On Fri, 25 Jan 2019 at 16:44, M. Manna <ma...@gmail.com> wrote:
>
> > Hello,
> >
> > I am trying to revive this thread. I only got 1 binding vote so far.
> >
> > Please feel free to revisit and comment here.
> >
> > Thanks,
> >
> > On Thu, 25 Oct 2018 at 00:15, M. Manna <ma...@gmail.com> wrote:
> >
> >> Hey IJ,
> >>
> >> Thanks for your interest in the KIP.
> >>
> >> My point was simply that the round-robin should happen even if the key is
> >> not null. As for the importance of key in our case, we treat the key as
> >> metadata. Each key is composed of certain info which are parsed by our
> >> consumer thread. We will then determine whether it's an actionable message
> >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> >> append this metadata with the record and parse it there?". But that means
> >> the following:
> >>
> >> 1) I'm always passing null key to achieve this - I would like to pass
> >> Null/Not-Null/Other key i.e. flexibility
> >> 2) Suppose the message size is 99 KB and and max message bytes allowed is
> >> 100K. Now prefixing metadata with message results into the actual message
> >> being 101K. This will fail at producer level and cause a retry/log this in
> >> our DB for future pickup.
> >>
> >> To avoid all these, we are simply proposing this new partitioner class.
> >> but all Kafka new releases will still have DefaultPartitioner as default,
> >> unless they change the prop file to use our new class.
> >>
> >> Regards,
> >>
> >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma <is...@juma.me.uk> wrote:
> >>
> >>> Thanks for the KIP. Can you please elaborate on the need for the key in
> >>> this case? The KIP simply states that the key is needed for metadata, but
> >>> doesn't give any more details.
> >>>
> >>> Ismael
> >>>
> >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna <ma...@gmail.com> wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > I have made necessary changes as per the original discussion thread,
> >>> and
> >>> > would like to put it for votes.
> >>> >
> >>> > Thank you very much for your suggestion and guidance so far.
> >>> >
> >>> > Regards,
> >>> >
> >>>
> >>
>