You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Anna Povzner <an...@confluent.io> on 2020/08/11 02:27:49 UTC
Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers
Hi All,
I wanted to let everyone know that we would like to make the following
changes to the KIP:
1.
Expose connection acceptance rate metrics (broker-wide and per-listener)
and per-listener average throttle time metrics for better observability and
debugging.
2.
KIP-599 introduced a new implementation of MeasurableStat that
implements a token bucket, which improves rate throttling for bursty
workloads (KAFKA-10162). We would like to use this same mechanism for
connection accept rate throttling.
I updated the KIP to reflect these changes.
Let me know if you have any concerns.
Thanks,
Anna
On Thu, May 21, 2020 at 5:42 PM Anna Povzner <an...@confluent.io> wrote:
> The vote for KIP-612 has passed with 3 binding and 3 non-binding +1s, and
> no objections.
>
>
> Thanks everyone for reviews and feedback,
>
> Anna
>
> On Tue, May 19, 2020 at 2:41 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> +1 (binding)
>>
>> Thanks for the KIP, Anna!
>>
>> Regards,
>>
>> Rajini
>>
>>
>> On Tue, May 19, 2020 at 9:32 AM Alexandre Dupriez <
>> alexandre.dupriez@gmail.com> wrote:
>>
>> > +1 (non-binding)
>> >
>> > Thank you for the KIP!
>> >
>> >
>> > Le mar. 19 mai 2020 à 07:57, David Jacot <dj...@confluent.io> a écrit
>> :
>> > >
>> > > +1 (non-binding)
>> > >
>> > > Thanks for the KIP, Anna!
>> > >
>> > > On Tue, May 19, 2020 at 7:12 AM Satish Duggana <
>> satish.duggana@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > +1 (non-binding)
>> > > > Thanks Anna for the nice feature to control the connection creation
>> > rate
>> > > > from the clients.
>> > > >
>> > > > On Tue, May 19, 2020 at 8:16 AM Gwen Shapira <gw...@confluent.io>
>> > wrote:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > Thank you for driving this, Anna
>> > > > >
>> > > > > On Mon, May 18, 2020 at 4:55 PM Anna Povzner <an...@confluent.io>
>> > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > I would like to start the vote on KIP-612: Ability to limit
>> > connection
>> > > > > > creation rate on brokers.
>> > > > > >
>> > > > > > For reference, here is the KIP wiki:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers
>> > > > > >
>> > > > > > And discussion thread:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>> https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Anna
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Gwen Shapira
>> > > > > Engineering Manager | Confluent
>> > > > > 650.450.2760 | @gwenshap
>> > > > > Follow us: Twitter | blog
>> > > > >
>> > > >
>> >
>>
>
Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers
Posted by Anna Povzner <an...@confluent.io>.
Hi David,
You are right, using a token bucket exposes the metric for the number of
remaining tokens in the bucket. Since we also want to evaluate using a
token bucket for bandwidth & request throttling, it would be better to have
this discussion separately in a separate KIP. This KIP does not implement
token bucket throttling, and I made sure that KIP wiki does not mention it
as well. Hopefully, I haven't added any confusion.
As David Mao is implementing per-IP part of this KIP, he will update the
KIP wiki and notify this thread with a couple of more details that were not
mentioned in the KIP wiki (e.g., per-IP rate metrics exposed when
throttling).
Thanks,
Anna
On Wed, Sep 2, 2020 at 1:06 PM David Jacot <dj...@confluent.io> wrote:
> Hi Anna,
>
> Thanks for the update.
>
> If you use Token Bucket, it will expose another metric which reports the
> number of remaining tokens in the bucket, in addition to the current rate
> metric. It would be great to add it in the metrics section of the KIP as
> well
> for completeness.
>
> Best,
> David
>
> On Tue, Aug 11, 2020 at 4:28 AM Anna Povzner <an...@confluent.io> wrote:
>
> > Hi All,
> >
> > I wanted to let everyone know that we would like to make the following
> > changes to the KIP:
> >
> > 1.
> >
> > Expose connection acceptance rate metrics (broker-wide and
> per-listener)
> > and per-listener average throttle time metrics for better
> observability
> > and
> > debugging.
> > 2.
> >
> > KIP-599 introduced a new implementation of MeasurableStat that
> > implements a token bucket, which improves rate throttling for bursty
> > workloads (KAFKA-10162). We would like to use this same mechanism for
> > connection accept rate throttling.
> >
> >
> > I updated the KIP to reflect these changes.
> >
> > Let me know if you have any concerns.
> >
> > Thanks,
> >
> > Anna
> >
> >
> > On Thu, May 21, 2020 at 5:42 PM Anna Povzner <an...@confluent.io> wrote:
> >
> > > The vote for KIP-612 has passed with 3 binding and 3 non-binding +1s,
> and
> > > no objections.
> > >
> > >
> > > Thanks everyone for reviews and feedback,
> > >
> > > Anna
> > >
> > > On Tue, May 19, 2020 at 2:41 AM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks for the KIP, Anna!
> > >>
> > >> Regards,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Tue, May 19, 2020 at 9:32 AM Alexandre Dupriez <
> > >> alexandre.dupriez@gmail.com> wrote:
> > >>
> > >> > +1 (non-binding)
> > >> >
> > >> > Thank you for the KIP!
> > >> >
> > >> >
> > >> > Le mar. 19 mai 2020 à 07:57, David Jacot <dj...@confluent.io> a
> > écrit
> > >> :
> > >> > >
> > >> > > +1 (non-binding)
> > >> > >
> > >> > > Thanks for the KIP, Anna!
> > >> > >
> > >> > > On Tue, May 19, 2020 at 7:12 AM Satish Duggana <
> > >> satish.duggana@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > +1 (non-binding)
> > >> > > > Thanks Anna for the nice feature to control the connection
> > creation
> > >> > rate
> > >> > > > from the clients.
> > >> > > >
> > >> > > > On Tue, May 19, 2020 at 8:16 AM Gwen Shapira <gwen@confluent.io
> >
> > >> > wrote:
> > >> > > >
> > >> > > > > +1 (binding)
> > >> > > > >
> > >> > > > > Thank you for driving this, Anna
> > >> > > > >
> > >> > > > > On Mon, May 18, 2020 at 4:55 PM Anna Povzner <
> anna@confluent.io
> > >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Hi All,
> > >> > > > > >
> > >> > > > > > I would like to start the vote on KIP-612: Ability to limit
> > >> > connection
> > >> > > > > > creation rate on brokers.
> > >> > > > > >
> > >> > > > > > For reference, here is the KIP wiki:
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers
> > >> > > > > >
> > >> > > > > > And discussion thread:
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Anna
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Gwen Shapira
> > >> > > > > Engineering Manager | Confluent
> > >> > > > > 650.450.2760 | @gwenshap
> > >> > > > > Follow us: Twitter | blog
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers
Posted by David Jacot <dj...@confluent.io>.
Hi Anna,
Thanks for the update.
If you use Token Bucket, it will expose another metric which reports the
number of remaining tokens in the bucket, in addition to the current rate
metric. It would be great to add it in the metrics section of the KIP as
well
for completeness.
Best,
David
On Tue, Aug 11, 2020 at 4:28 AM Anna Povzner <an...@confluent.io> wrote:
> Hi All,
>
> I wanted to let everyone know that we would like to make the following
> changes to the KIP:
>
> 1.
>
> Expose connection acceptance rate metrics (broker-wide and per-listener)
> and per-listener average throttle time metrics for better observability
> and
> debugging.
> 2.
>
> KIP-599 introduced a new implementation of MeasurableStat that
> implements a token bucket, which improves rate throttling for bursty
> workloads (KAFKA-10162). We would like to use this same mechanism for
> connection accept rate throttling.
>
>
> I updated the KIP to reflect these changes.
>
> Let me know if you have any concerns.
>
> Thanks,
>
> Anna
>
>
> On Thu, May 21, 2020 at 5:42 PM Anna Povzner <an...@confluent.io> wrote:
>
> > The vote for KIP-612 has passed with 3 binding and 3 non-binding +1s, and
> > no objections.
> >
> >
> > Thanks everyone for reviews and feedback,
> >
> > Anna
> >
> > On Tue, May 19, 2020 at 2:41 AM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> >> +1 (binding)
> >>
> >> Thanks for the KIP, Anna!
> >>
> >> Regards,
> >>
> >> Rajini
> >>
> >>
> >> On Tue, May 19, 2020 at 9:32 AM Alexandre Dupriez <
> >> alexandre.dupriez@gmail.com> wrote:
> >>
> >> > +1 (non-binding)
> >> >
> >> > Thank you for the KIP!
> >> >
> >> >
> >> > Le mar. 19 mai 2020 à 07:57, David Jacot <dj...@confluent.io> a
> écrit
> >> :
> >> > >
> >> > > +1 (non-binding)
> >> > >
> >> > > Thanks for the KIP, Anna!
> >> > >
> >> > > On Tue, May 19, 2020 at 7:12 AM Satish Duggana <
> >> satish.duggana@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > +1 (non-binding)
> >> > > > Thanks Anna for the nice feature to control the connection
> creation
> >> > rate
> >> > > > from the clients.
> >> > > >
> >> > > > On Tue, May 19, 2020 at 8:16 AM Gwen Shapira <gw...@confluent.io>
> >> > wrote:
> >> > > >
> >> > > > > +1 (binding)
> >> > > > >
> >> > > > > Thank you for driving this, Anna
> >> > > > >
> >> > > > > On Mon, May 18, 2020 at 4:55 PM Anna Povzner <anna@confluent.io
> >
> >> > wrote:
> >> > > > >
> >> > > > > > Hi All,
> >> > > > > >
> >> > > > > > I would like to start the vote on KIP-612: Ability to limit
> >> > connection
> >> > > > > > creation rate on brokers.
> >> > > > > >
> >> > > > > > For reference, here is the KIP wiki:
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers
> >> > > > > >
> >> > > > > > And discussion thread:
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> >
> >>
> https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Anna
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Gwen Shapira
> >> > > > > Engineering Manager | Confluent
> >> > > > > 650.450.2760 | @gwenshap
> >> > > > > Follow us: Twitter | blog
> >> > > > >
> >> > > >
> >> >
> >>
> >
>