You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Gokul Ramanan Subramanian <go...@gmail.com> on 2020/07/21 13:07:29 UTC
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi.
Can we resume the voting process for KIP-578? I have addressed additional
comments by Boyang and Ismael.
Thanks.
On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian <
gokul2411s@gmail.com> wrote:
> Hi. Can we resume the voting process for KIP-578? Thanks.
>
> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian <
> gokul2411s@gmail.com> wrote:
>
>> Thanks Colin. Have updated the KIP per your recommendations. Let me know
>> what you think.
>>
>> Thanks Harsha for the vote.
>>
>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe <cm...@apache.org> wrote:
>>
>>> Hi Gokul Ramanan Subramanian,
>>>
>>> Thanks for the KIP.
>>>
>>> Can you please modify the KIP to remove the reference to the deprecated
>>> --zookeeper flag? This is not how kafka-configs.sh is supposed to be used
>>> in new versions of Kafka. You get a warning message if you do use this
>>> deprecated flag. As described in KIP-604, we are removing the --zookeeper
>>> flag in the Kafka 3.0 release. It also causes problems when people use the
>>> deprecated access mode-- for example, as you note in this KIP, it bypasses
>>> resource limits such as the ones described here.
>>>
>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about
>>> RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed
>>> message about which resource limit was hit (per broker limit, per cluster
>>> limit, whatever.) It would also be good to spell out that
>>> CreateTopicsPolicy plugins can also throw this exception, for consistency.
>>>
>>> I realize that 2 billion partitions seems like a very big number.
>>> However, filesystems have had to transition to 64 bit inode numbers as time
>>> has gone on. There doesn't seem to be any performance reason why this
>>> should be a 31 bit number, so let's just make these configurations longs,
>>> not ints.
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
>>> > Thanks for the KIP Gokul. This will be really useful for our use cases
>>> as
>>> > well.
>>> > +1 (binding).
>>> >
>>> > -Harsha
>>> >
>>> >
>>> > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian <
>>> > gokul2411s@gmail.com> wrote:
>>> >
>>> > > Hi.
>>> > >
>>> > > Any votes for this?
>>> > >
>>> > > Thanks.
>>> > >
>>> > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
>>> gokul2411s@
>>> > > gmail.com> wrote:
>>> > >
>>> > > Hello,
>>> > >
>>> > > I'd like to initialize voting on KIP-578:
>>> > > https://cwiki.apache.org/confluence/display/KAFKA/
>>> > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions
>>> > > .
>>> > >
>>> > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
>>> and Tom
>>> > > Bentley on the discussion thread. I have addressed their comments. I
>>> want
>>> > > to thank them for their time.
>>> > >
>>> > > If there are any more concerns about the KIP, I am happy to discuss
>>> them
>>> > > further.
>>> > >
>>> > > Thanks.
>>> > >
>>> > >
>>> >
>>>
>>
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Posted by Gokul Ramanan Subramanian <go...@gmail.com>.
Hi, at this point, this KIP has one binding vote from Harsha.
Does anyone else have an opinion on this KIP? Do you think we should go
forward with this KIP once we release Kafka 3.0, since there wouldn't be
the ZK backdoor in 3.0, making the partition limits properly enforceable?
Thanks.
On Wed, Aug 5, 2020 at 5:59 PM Harsha Ch <ha...@gmail.com> wrote:
> Thanks for the updates Gokul. +1 (binding).
>
> Thanks,
>
> Harsha
>
> On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian <
> gokul2411s@gmail.com > wrote:
>
> >
> >
> >
> > Hi.
> >
> >
> >
> > I request the community to kindly resume the voting process for KIP-578.
> > If you have further comments, we can address those as well.
> >
> >
> >
> > Thanks and cheers.
> >
> >
> >
> > On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul2411s@
> gmail.
> > com ( gokul2411s@gmail.com ) > wrote:
> >
> >
> >>
> >>
> >> Hi.
> >>
> >>
> >>
> >> Can we resume the voting process for KIP-578? I have addressed
> additional
> >> comments by Boyang and Ismael.
> >>
> >>
> >>
> >> Thanks.
> >>
> >>
> >>
> >> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul2411s@
> gmail.
> >> com ( gokul2411s@gmail.com ) > wrote:
> >>
> >>
> >>>
> >>>
> >>> Hi. Can we resume the voting process for KIP-578? Thanks.
> >>>
> >>>
> >>>
> >>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul2411s@
> gmail.
> >>> com ( gokul2411s@gmail.com ) > wrote:
> >>>
> >>>
> >>>>
> >>>>
> >>>> Thanks Colin. Have updated the KIP per your recommendations. Let me
> know
> >>>> what you think.
> >>>>
> >>>>
> >>>>
> >>>> Thanks Harsha for the vote.
> >>>>
> >>>>
> >>>>
> >>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe < cmccabe@ apache. org (
> >>>> cmccabe@apache.org ) > wrote:
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> Hi Gokul Ramanan Subramanian,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks for the KIP.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Can you please modify the KIP to remove the reference to the
> deprecated
> >>>>> --zookeeper flag? This is not how kafka-configs. sh (
> >>>>> http://kafka-configs.sh/ ) is supposed to be used in new versions
> of Kafka.
> >>>>> You get a warning message if you do use this deprecated flag. As
> described
> >>>>> in KIP-604, we are removing the --zookeeper flag in the Kafka 3.0
> release.
> >>>>> It also causes problems when people use the deprecated access mode--
> for
> >>>>> example, as you note in this KIP, it bypasses resource limits such
> as the
> >>>>> ones described here.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about
> RESOURCE_LIMIT_REACHED?
> >>>>> Then the error string can contain the detailed message about which
> >>>>> resource limit was hit (per broker limit, per cluster limit,
> whatever.) It
> >>>>> would also be good to spell out that CreateTopicsPolicy plugins can
> also
> >>>>> throw this exception, for consistency.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I realize that 2 billion partitions seems like a very big number.
> However,
> >>>>> filesystems have had to transition to 64 bit inode numbers as time
> has
> >>>>> gone on. There doesn't seem to be any performance reason why this
> should
> >>>>> be a 31 bit number, so let's just make these configurations longs,
> not
> >>>>> ints.
> >>>>>
> >>>>>
> >>>>>
> >>>>> best,
> >>>>> Colin
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Thanks for the KIP Gokul. This will be really useful for our use
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> cases as
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> well.
> >>>>>> +1 (binding).
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -Harsha
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian <
> gokul2411s@ gmail.
> >>>>>> com ( gokul2411s@gmail.com ) > wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Any votes for this?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> gokul2411s@
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> gmail. com ( http://gmail.com/ ) > wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I'd like to initialize voting on KIP-578:
> >>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
> >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ )
> KIP-578%3A+Add+configuration+to+limit+number+of+partitions
> >>>>>>>
> >>>>>>> .
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> and Tom
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Bentley on the discussion thread. I have addressed their comments.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> I want
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> to thank them for their time.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> If there are any more concerns about the KIP, I am happy to discuss
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> them
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> further.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
> >
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Posted by Harsha Ch <ha...@gmail.com>.
Thanks for the updates Gokul. +1 (binding).
Thanks,
Harsha
On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian < gokul2411s@gmail.com > wrote:
>
>
>
> Hi.
>
>
>
> I request the community to kindly resume the voting process for KIP-578.
> If you have further comments, we can address those as well.
>
>
>
> Thanks and cheers.
>
>
>
> On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul2411s@ gmail.
> com ( gokul2411s@gmail.com ) > wrote:
>
>
>>
>>
>> Hi.
>>
>>
>>
>> Can we resume the voting process for KIP-578? I have addressed additional
>> comments by Boyang and Ismael.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul2411s@ gmail.
>> com ( gokul2411s@gmail.com ) > wrote:
>>
>>
>>>
>>>
>>> Hi. Can we resume the voting process for KIP-578? Thanks.
>>>
>>>
>>>
>>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul2411s@ gmail.
>>> com ( gokul2411s@gmail.com ) > wrote:
>>>
>>>
>>>>
>>>>
>>>> Thanks Colin. Have updated the KIP per your recommendations. Let me know
>>>> what you think.
>>>>
>>>>
>>>>
>>>> Thanks Harsha for the vote.
>>>>
>>>>
>>>>
>>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe < cmccabe@ apache. org (
>>>> cmccabe@apache.org ) > wrote:
>>>>
>>>>
>>>>>
>>>>>
>>>>> Hi Gokul Ramanan Subramanian,
>>>>>
>>>>>
>>>>>
>>>>> Thanks for the KIP.
>>>>>
>>>>>
>>>>>
>>>>> Can you please modify the KIP to remove the reference to the deprecated
>>>>> --zookeeper flag? This is not how kafka-configs. sh (
>>>>> http://kafka-configs.sh/ ) is supposed to be used in new versions of Kafka.
>>>>> You get a warning message if you do use this deprecated flag. As described
>>>>> in KIP-604, we are removing the --zookeeper flag in the Kafka 3.0 release.
>>>>> It also causes problems when people use the deprecated access mode-- for
>>>>> example, as you note in this KIP, it bypasses resource limits such as the
>>>>> ones described here.
>>>>>
>>>>>
>>>>>
>>>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about RESOURCE_LIMIT_REACHED?
>>>>> Then the error string can contain the detailed message about which
>>>>> resource limit was hit (per broker limit, per cluster limit, whatever.) It
>>>>> would also be good to spell out that CreateTopicsPolicy plugins can also
>>>>> throw this exception, for consistency.
>>>>>
>>>>>
>>>>>
>>>>> I realize that 2 billion partitions seems like a very big number. However,
>>>>> filesystems have had to transition to 64 bit inode numbers as time has
>>>>> gone on. There doesn't seem to be any performance reason why this should
>>>>> be a 31 bit number, so let's just make these configurations longs, not
>>>>> ints.
>>>>>
>>>>>
>>>>>
>>>>> best,
>>>>> Colin
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks for the KIP Gokul. This will be really useful for our use
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> cases as
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> well.
>>>>>> +1 (binding).
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Harsha
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < gokul2411s@ gmail.
>>>>>> com ( gokul2411s@gmail.com ) > wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any votes for this?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> gokul2411s@
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> gmail. com ( http://gmail.com/ ) > wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I'd like to initialize voting on KIP-578:
>>>>>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) KIP-578%3A+Add+configuration+to+limit+number+of+partitions
>>>>>>>
>>>>>>> .
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> and Tom
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Bentley on the discussion thread. I have addressed their comments.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I want
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> to thank them for their time.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If there are any more concerns about the KIP, I am happy to discuss
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> them
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> further.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Posted by Gokul Ramanan Subramanian <go...@gmail.com>.
Hi.
I request the community to kindly resume the voting process for KIP-578. If
you have further comments, we can address those as well.
Thanks and cheers.
On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian <
gokul2411s@gmail.com> wrote:
> Hi.
>
> Can we resume the voting process for KIP-578? I have addressed additional
> comments by Boyang and Ismael.
>
> Thanks.
>
> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian <
> gokul2411s@gmail.com> wrote:
>
>> Hi. Can we resume the voting process for KIP-578? Thanks.
>>
>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian <
>> gokul2411s@gmail.com> wrote:
>>
>>> Thanks Colin. Have updated the KIP per your recommendations. Let me know
>>> what you think.
>>>
>>> Thanks Harsha for the vote.
>>>
>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe <cm...@apache.org> wrote:
>>>
>>>> Hi Gokul Ramanan Subramanian,
>>>>
>>>> Thanks for the KIP.
>>>>
>>>> Can you please modify the KIP to remove the reference to the deprecated
>>>> --zookeeper flag? This is not how kafka-configs.sh is supposed to be used
>>>> in new versions of Kafka. You get a warning message if you do use this
>>>> deprecated flag. As described in KIP-604, we are removing the --zookeeper
>>>> flag in the Kafka 3.0 release. It also causes problems when people use the
>>>> deprecated access mode-- for example, as you note in this KIP, it bypasses
>>>> resource limits such as the ones described here.
>>>>
>>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about
>>>> RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed
>>>> message about which resource limit was hit (per broker limit, per cluster
>>>> limit, whatever.) It would also be good to spell out that
>>>> CreateTopicsPolicy plugins can also throw this exception, for consistency.
>>>>
>>>> I realize that 2 billion partitions seems like a very big number.
>>>> However, filesystems have had to transition to 64 bit inode numbers as time
>>>> has gone on. There doesn't seem to be any performance reason why this
>>>> should be a 31 bit number, so let's just make these configurations longs,
>>>> not ints.
>>>>
>>>> best,
>>>> Colin
>>>>
>>>>
>>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
>>>> > Thanks for the KIP Gokul. This will be really useful for our use
>>>> cases as
>>>> > well.
>>>> > +1 (binding).
>>>> >
>>>> > -Harsha
>>>> >
>>>> >
>>>> > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian <
>>>> > gokul2411s@gmail.com> wrote:
>>>> >
>>>> > > Hi.
>>>> > >
>>>> > > Any votes for this?
>>>> > >
>>>> > > Thanks.
>>>> > >
>>>> > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
>>>> gokul2411s@
>>>> > > gmail.com> wrote:
>>>> > >
>>>> > > Hello,
>>>> > >
>>>> > > I'd like to initialize voting on KIP-578:
>>>> > > https://cwiki.apache.org/confluence/display/KAFKA/
>>>> > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions
>>>> > > .
>>>> > >
>>>> > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
>>>> and Tom
>>>> > > Bentley on the discussion thread. I have addressed their comments.
>>>> I want
>>>> > > to thank them for their time.
>>>> > >
>>>> > > If there are any more concerns about the KIP, I am happy to discuss
>>>> them
>>>> > > further.
>>>> > >
>>>> > > Thanks.
>>>> > >
>>>> > >
>>>> >
>>>>
>>>