Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers

[Anna Povzner <an...@confluent.io>]

Hi David,

You are right, using a token bucket exposes the metric for the number of
remaining tokens in the bucket. Since we also want to evaluate using a
token bucket for bandwidth & request throttling, it would be better to have
this discussion separately in a separate KIP. This KIP does not implement
token bucket throttling, and I made sure that KIP wiki does not mention it
as well. Hopefully, I haven't added any confusion.

As David Mao is implementing per-IP part of this KIP, he will update the
KIP wiki and notify this thread with a couple of more details that were not
mentioned in the KIP wiki (e.g., per-IP rate metrics exposed when
throttling).

Thanks,
Anna

On Wed, Sep 2, 2020 at 1:06 PM David Jacot <dj...@confluent.io> wrote:

> Hi Anna,
>
> Thanks for the update.
>
> If you use Token Bucket, it will expose another metric which reports the
> number of remaining tokens in the bucket, in addition to the current rate
> metric. It would be great to add it in the metrics section of the KIP as
> well
> for completeness.
>
> Best,
> David
>
> On Tue, Aug 11, 2020 at 4:28 AM Anna Povzner <an...@confluent.io> wrote:
>
> > Hi All,
> >
> > I wanted to let everyone know that we would like to make the following
> > changes to the KIP:
> >
> >    1.
> >
> >    Expose connection acceptance rate metrics (broker-wide and
> per-listener)
> >    and per-listener average throttle time metrics for better
> observability
> > and
> >    debugging.
> >    2.
> >
> >    KIP-599 introduced a new implementation of MeasurableStat that
> >    implements a token bucket, which improves rate throttling for bursty
> >    workloads (KAFKA-10162). We would like to use this same mechanism for
> >    connection accept rate throttling.
> >
> >
> > I updated the KIP to reflect these changes.
> >
> > Let me know if you have any concerns.
> >
> > Thanks,
> >
> > Anna
> >
> >
> > On Thu, May 21, 2020 at 5:42 PM Anna Povzner <an...@confluent.io> wrote:
> >
> > > The vote for KIP-612 has passed with 3 binding and 3 non-binding +1s,
> and
> > > no objections.
> > >
> > >
> > > Thanks everyone for reviews and feedback,
> > >
> > > Anna
> > >
> > > On Tue, May 19, 2020 at 2:41 AM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks for the KIP, Anna!
> > >>
> > >> Regards,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Tue, May 19, 2020 at 9:32 AM Alexandre Dupriez <
> > >> alexandre.dupriez@gmail.com> wrote:
> > >>
> > >> > +1 (non-binding)
> > >> >
> > >> > Thank you for the KIP!
> > >> >
> > >> >
> > >> > Le mar. 19 mai 2020 à 07:57, David Jacot <dj...@confluent.io> a
> > écrit
> > >> :
> > >> > >
> > >> > > +1 (non-binding)
> > >> > >
> > >> > > Thanks for the KIP, Anna!
> > >> > >
> > >> > > On Tue, May 19, 2020 at 7:12 AM Satish Duggana <
> > >> satish.duggana@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > +1 (non-binding)
> > >> > > > Thanks Anna for the nice feature to control the connection
> > creation
> > >> > rate
> > >> > > > from the clients.
> > >> > > >
> > >> > > > On Tue, May 19, 2020 at 8:16 AM Gwen Shapira <gwen@confluent.io
> >
> > >> > wrote:
> > >> > > >
> > >> > > > > +1 (binding)
> > >> > > > >
> > >> > > > > Thank you for driving this, Anna
> > >> > > > >
> > >> > > > > On Mon, May 18, 2020 at 4:55 PM Anna Povzner <
> anna@confluent.io
> > >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Hi All,
> > >> > > > > >
> > >> > > > > > I would like to start the vote on KIP-612: Ability to limit
> > >> > connection
> > >> > > > > > creation rate on brokers.
> > >> > > > > >
> > >> > > > > > For reference, here is the KIP wiki:
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-612%3A+Ability+to+Limit+Connection+Creation+Rate+on+Brokers
> > >> > > > > >
> > >> > > > > > And discussion thread:
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/r61162661fa307d0bc5c8326818bf223a689c49e1c828c9928ee26969%40%3Cdev.kafka.apache.org%3E
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > Anna
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Gwen Shapira
> > >> > > > > Engineering Manager | Confluent
> > >> > > > > 650.450.2760 | @gwenshap
> > >> > > > > Follow us: Twitter | blog
> > >> > > > >
> > >> > > >
> > >> >
> > >>
> > >
> >
>