You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Mocheng Guo <gm...@gmail.com> on 2019/07/16 19:46:36 UTC
complete quota system proposal
Hi, currently Zookeeper has a storage quota feature which is informational
only and there are a few JIRAs to enforce throttling.
ZOOKEEPER-451
ZOOKEEPER-2593
ZOOKEEPER-3301
I am wondering if a more complete quota/throttling system covering metrics
such as storage/rate/watch/connection would be valuable here? We at FB have
been battling with excessive system usage from clients and did some work in
this area. We'd be happy to submit the work for review and consolidate with
existing efforts if people feel this is a good feature to add.
thanks
Mocheng
Re: complete quota system proposal
Posted by Mocheng Guo <gm...@gmail.com>.
Good point and I missed it. Yes client identity inside certificate can be
used if connection is secured, otherwise we could use existing zk auth
protocol to enable client sending its id to server.
0. client identification
1. quota configuration - metric key and value, format, storage
2. metrics collection and export - storage/rate/watch/connection
3. throttling implementation based on metrics inside server/client
On Thu, Jul 18, 2019 at 9:18 PM Michael Han <ha...@apache.org> wrote:
> Sounds good. I think we also need a plan on how to identify clients. If
> auth is enabled, we might be able to reuse some of the auth information,
> but a generalized client-id based approach sounds better.
>
> On Thu, Jul 18, 2019 at 11:32 AM Mocheng Guo <gm...@gmail.com> wrote:
>
> > Hi Michael, thanks for your feedback and I will create an epic for this.
> I
> > will divide into following stories and please let me know if this makes
> > sense
> >
> > 1. quota configuration - metric key and value, format, storage
> > 2. metrics collection and export - storage/rate/watch/connection
> > 3. throttling implementation based on metrics inside server/client
> >
> > On Wed, Jul 17, 2019 at 8:02 PM Michael Han <ha...@apache.org> wrote:
> >
> > > >> a more complete quota/throttling system would be valuable here
> > >
> > > Absolutely. This will improve the stability of a shared zookeeper
> > cluster,
> > > which is a very common use case. Because the lack of enforce quota (and
> > > other soft isolation mechanisms such as flow control / request
> limiting),
> > > the usual practice of dealing with such excessive client usages is to
> > > provision dedicated ZK cluster per customer which creates additional
> > > operation overheads and is waste of resource.
> > >
> > > >> We'd be happy to submit the work for review
> > >
> > > Sounds great. I would suggest we create an epic around this and then
> > break
> > > down the stories to multiple sub tasks and move existing issues that's
> > > relevant under the epic, so we can consolidate efforts and have a
> single
> > > place to track the progress.
> > >
> > >
> > > On Tue, Jul 16, 2019 at 12:46 PM Mocheng Guo <gm...@gmail.com>
> wrote:
> > >
> > > > Hi, currently Zookeeper has a storage quota feature which is
> > > informational
> > > > only and there are a few JIRAs to enforce throttling.
> > > >
> > > > ZOOKEEPER-451
> > > > ZOOKEEPER-2593
> > > > ZOOKEEPER-3301
> > > >
> > > > I am wondering if a more complete quota/throttling system covering
> > > metrics
> > > > such as storage/rate/watch/connection would be valuable here? We at
> FB
> > > have
> > > > been battling with excessive system usage from clients and did some
> > work
> > > in
> > > > this area. We'd be happy to submit the work for review and
> consolidate
> > > with
> > > > existing efforts if people feel this is a good feature to add.
> > > >
> > > > thanks
> > > > Mocheng
> > > >
> > >
> >
>
Re: complete quota system proposal
Posted by Michael Han <ha...@apache.org>.
Sounds good. I think we also need a plan on how to identify clients. If
auth is enabled, we might be able to reuse some of the auth information,
but a generalized client-id based approach sounds better.
On Thu, Jul 18, 2019 at 11:32 AM Mocheng Guo <gm...@gmail.com> wrote:
> Hi Michael, thanks for your feedback and I will create an epic for this. I
> will divide into following stories and please let me know if this makes
> sense
>
> 1. quota configuration - metric key and value, format, storage
> 2. metrics collection and export - storage/rate/watch/connection
> 3. throttling implementation based on metrics inside server/client
>
> On Wed, Jul 17, 2019 at 8:02 PM Michael Han <ha...@apache.org> wrote:
>
> > >> a more complete quota/throttling system would be valuable here
> >
> > Absolutely. This will improve the stability of a shared zookeeper
> cluster,
> > which is a very common use case. Because the lack of enforce quota (and
> > other soft isolation mechanisms such as flow control / request limiting),
> > the usual practice of dealing with such excessive client usages is to
> > provision dedicated ZK cluster per customer which creates additional
> > operation overheads and is waste of resource.
> >
> > >> We'd be happy to submit the work for review
> >
> > Sounds great. I would suggest we create an epic around this and then
> break
> > down the stories to multiple sub tasks and move existing issues that's
> > relevant under the epic, so we can consolidate efforts and have a single
> > place to track the progress.
> >
> >
> > On Tue, Jul 16, 2019 at 12:46 PM Mocheng Guo <gm...@gmail.com> wrote:
> >
> > > Hi, currently Zookeeper has a storage quota feature which is
> > informational
> > > only and there are a few JIRAs to enforce throttling.
> > >
> > > ZOOKEEPER-451
> > > ZOOKEEPER-2593
> > > ZOOKEEPER-3301
> > >
> > > I am wondering if a more complete quota/throttling system covering
> > metrics
> > > such as storage/rate/watch/connection would be valuable here? We at FB
> > have
> > > been battling with excessive system usage from clients and did some
> work
> > in
> > > this area. We'd be happy to submit the work for review and consolidate
> > with
> > > existing efforts if people feel this is a good feature to add.
> > >
> > > thanks
> > > Mocheng
> > >
> >
>
Re: complete quota system proposal
Posted by Mocheng Guo <gm...@gmail.com>.
Hi Michael, thanks for your feedback and I will create an epic for this. I
will divide into following stories and please let me know if this makes
sense
1. quota configuration - metric key and value, format, storage
2. metrics collection and export - storage/rate/watch/connection
3. throttling implementation based on metrics inside server/client
On Wed, Jul 17, 2019 at 8:02 PM Michael Han <ha...@apache.org> wrote:
> >> a more complete quota/throttling system would be valuable here
>
> Absolutely. This will improve the stability of a shared zookeeper cluster,
> which is a very common use case. Because the lack of enforce quota (and
> other soft isolation mechanisms such as flow control / request limiting),
> the usual practice of dealing with such excessive client usages is to
> provision dedicated ZK cluster per customer which creates additional
> operation overheads and is waste of resource.
>
> >> We'd be happy to submit the work for review
>
> Sounds great. I would suggest we create an epic around this and then break
> down the stories to multiple sub tasks and move existing issues that's
> relevant under the epic, so we can consolidate efforts and have a single
> place to track the progress.
>
>
> On Tue, Jul 16, 2019 at 12:46 PM Mocheng Guo <gm...@gmail.com> wrote:
>
> > Hi, currently Zookeeper has a storage quota feature which is
> informational
> > only and there are a few JIRAs to enforce throttling.
> >
> > ZOOKEEPER-451
> > ZOOKEEPER-2593
> > ZOOKEEPER-3301
> >
> > I am wondering if a more complete quota/throttling system covering
> metrics
> > such as storage/rate/watch/connection would be valuable here? We at FB
> have
> > been battling with excessive system usage from clients and did some work
> in
> > this area. We'd be happy to submit the work for review and consolidate
> with
> > existing efforts if people feel this is a good feature to add.
> >
> > thanks
> > Mocheng
> >
>
Re: complete quota system proposal
Posted by Michael Han <ha...@apache.org>.
>> a more complete quota/throttling system would be valuable here
Absolutely. This will improve the stability of a shared zookeeper cluster,
which is a very common use case. Because the lack of enforce quota (and
other soft isolation mechanisms such as flow control / request limiting),
the usual practice of dealing with such excessive client usages is to
provision dedicated ZK cluster per customer which creates additional
operation overheads and is waste of resource.
>> We'd be happy to submit the work for review
Sounds great. I would suggest we create an epic around this and then break
down the stories to multiple sub tasks and move existing issues that's
relevant under the epic, so we can consolidate efforts and have a single
place to track the progress.
On Tue, Jul 16, 2019 at 12:46 PM Mocheng Guo <gm...@gmail.com> wrote:
> Hi, currently Zookeeper has a storage quota feature which is informational
> only and there are a few JIRAs to enforce throttling.
>
> ZOOKEEPER-451
> ZOOKEEPER-2593
> ZOOKEEPER-3301
>
> I am wondering if a more complete quota/throttling system covering metrics
> such as storage/rate/watch/connection would be valuable here? We at FB have
> been battling with excessive system usage from clients and did some work in
> this area. We'd be happy to submit the work for review and consolidate with
> existing efforts if people feel this is a good feature to add.
>
> thanks
> Mocheng
>