You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Hang Chen <ch...@apache.org> on 2022/01/04 14:24:27 UTC
[DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
https://github.com/apache/pulsar/issues/13340
Pasted below for quoting convenience.
----
### Motivation
The ThresholdShedder load balance policy since Pulsar 2.6.0 by
https://github.com/apache/pulsar/pull/6772. It can resolve many load
balance issues of `OverloadShedder` and works well in many Pulsar
production clusters.
In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
policy is `OverloadShedder`.
I think it's a good time for 2.10 to change default load balance
policy to `ThresholdShedder`, it will make throughput more balance
between brokers.
### Proposed Changes
In 2.10 release，for `broker.conf`, change
`loadBalancerLoadSheddingStrategy` from
`org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
`org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Hang Chen <ch...@apache.org>.
Hi Michael,
Thanks for you review, it will push a PR to add a test for
`ThresholdShedder `.
Best,
Hang
Michael Marshall <mm...@apache.org> 于2022年1月5日周三 00:55写道：
>
> Hi Hang Chen,
>
> I support changing the default for 2.10.
>
> However, as far as I can tell, there are unit tests for the
> `OverloadShedder` class but not for the `ThresholdShedder` class. I
> think we should add unit tests before we change the default.
>
> Regarding integration tests, I assume that we implicitly test the
> default LoadShedder, but I haven't checked myself.
>
> Thanks,
> Michael
>
> On Tue, Jan 4, 2022 at 8:24 AM Hang Chen <ch...@apache.org> wrote:
> >
> > https://github.com/apache/pulsar/issues/13340
> >
> > Pasted below for quoting convenience.
> >
> > ----
> > ### Motivation
> > The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> > https://github.com/apache/pulsar/pull/6772. It can resolve many load
> > balance issues of `OverloadShedder` and works well in many Pulsar
> > production clusters.
> >
> > In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> > policy is `OverloadShedder`.
> >
> > I think it's a good time for 2.10 to change default load balance
> > policy to `ThresholdShedder`, it will make throughput more balance
> > between brokers.
> >
> > ### Proposed Changes
> > In 2.10 release，for `broker.conf`, change
> > `loadBalancerLoadSheddingStrategy` from
> > `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> > `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Michael Marshall <mm...@apache.org>.
Hi Hang Chen,
I support changing the default for 2.10.
However, as far as I can tell, there are unit tests for the
`OverloadShedder` class but not for the `ThresholdShedder` class. I
think we should add unit tests before we change the default.
Regarding integration tests, I assume that we implicitly test the
default LoadShedder, but I haven't checked myself.
Thanks,
Michael
On Tue, Jan 4, 2022 at 8:24 AM Hang Chen <ch...@apache.org> wrote:
>
> https://github.com/apache/pulsar/issues/13340
>
> Pasted below for quoting convenience.
>
> ----
> ### Motivation
> The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> https://github.com/apache/pulsar/pull/6772. It can resolve many load
> balance issues of `OverloadShedder` and works well in many Pulsar
> production clusters.
>
> In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> policy is `OverloadShedder`.
>
> I think it's a good time for 2.10 to change default load balance
> policy to `ThresholdShedder`, it will make throughput more balance
> between brokers.
>
> ### Proposed Changes
> In 2.10 release，for `broker.conf`, change
> `loadBalancerLoadSheddingStrategy` from
> `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Hang Chen <ch...@apache.org>.
Looks there is no objection, I will start the official vote for PIP-122
Regards,
Hang
Michael Marshall <mm...@apache.org> 于2022年1月8日周六 07:44写道：
>
> Hi Frank,
>
> You can configure the Load Shedding Strategy in the `broker.conf` file
> using `loadBalancerLoadSheddingStrategy`.
>
> The configuration is missing from the document you referenced--it
> should be there, though.
>
> There was a recent PR to add documentation describing the different
> configuration options: [0]. Those docs won't go live until 2.10.0 is
> released. We could probably add parts of that documentation to the
> historical docs, since the Shedding Strategy has existed for some
> time.
>
> - Michael
>
> [0] https://github.com/apache/pulsar/pull/12902/files
>
> On Fri, Jan 7, 2022 at 1:29 PM Frank Kelly <fk...@cogitocorp.com> wrote:
> >
> > This is very cool.
> >
> > How is `ThresholdShedder` chosen - I don't see if referenced here?
> > https://pulsar.apache.org/docs/en/reference-configuration/
> >
> > -F
> >
> > On Thu, Jan 6, 2022 at 7:45 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > > +1 ThresholdShedder is a much better policy
> > >
> > > On Tue, Jan 4, 2022 at 6:24 AM Hang Chen <ch...@apache.org> wrote:
> > >
> > > > https://github.com/apache/pulsar/issues/13340
> > > >
> > > > Pasted below for quoting convenience.
> > > >
> > > > ----
> > > > ### Motivation
> > > > The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> > > > https://github.com/apache/pulsar/pull/6772. It can resolve many load
> > > > balance issues of `OverloadShedder` and works well in many Pulsar
> > > > production clusters.
> > > >
> > > > In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> > > > policy is `OverloadShedder`.
> > > >
> > > > I think it's a good time for 2.10 to change default load balance
> > > > policy to `ThresholdShedder`, it will make throughput more balance
> > > > between brokers.
> > > >
> > > > ### Proposed Changes
> > > > In 2.10 release，for `broker.conf`, change
> > > > `loadBalancerLoadSheddingStrategy` from
> > > > `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> > > > `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
> > > >
> > >
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Michael Marshall <mm...@apache.org>.
Hi Frank,
You can configure the Load Shedding Strategy in the `broker.conf` file
using `loadBalancerLoadSheddingStrategy`.
The configuration is missing from the document you referenced--it
should be there, though.
There was a recent PR to add documentation describing the different
configuration options: [0]. Those docs won't go live until 2.10.0 is
released. We could probably add parts of that documentation to the
historical docs, since the Shedding Strategy has existed for some
time.
- Michael
[0] https://github.com/apache/pulsar/pull/12902/files
On Fri, Jan 7, 2022 at 1:29 PM Frank Kelly <fk...@cogitocorp.com> wrote:
>
> This is very cool.
>
> How is `ThresholdShedder` chosen - I don't see if referenced here?
> https://pulsar.apache.org/docs/en/reference-configuration/
>
> -F
>
> On Thu, Jan 6, 2022 at 7:45 PM Sijie Guo <gu...@gmail.com> wrote:
>
> > +1 ThresholdShedder is a much better policy
> >
> > On Tue, Jan 4, 2022 at 6:24 AM Hang Chen <ch...@apache.org> wrote:
> >
> > > https://github.com/apache/pulsar/issues/13340
> > >
> > > Pasted below for quoting convenience.
> > >
> > > ----
> > > ### Motivation
> > > The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> > > https://github.com/apache/pulsar/pull/6772. It can resolve many load
> > > balance issues of `OverloadShedder` and works well in many Pulsar
> > > production clusters.
> > >
> > > In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> > > policy is `OverloadShedder`.
> > >
> > > I think it's a good time for 2.10 to change default load balance
> > > policy to `ThresholdShedder`, it will make throughput more balance
> > > between brokers.
> > >
> > > ### Proposed Changes
> > > In 2.10 release，for `broker.conf`, change
> > > `loadBalancerLoadSheddingStrategy` from
> > > `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> > > `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
> > >
> >
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Frank Kelly <fk...@cogitocorp.com>.
This is very cool.
How is `ThresholdShedder` chosen - I don't see if referenced here?
https://pulsar.apache.org/docs/en/reference-configuration/
-F
On Thu, Jan 6, 2022 at 7:45 PM Sijie Guo <gu...@gmail.com> wrote:
> +1 ThresholdShedder is a much better policy
>
> On Tue, Jan 4, 2022 at 6:24 AM Hang Chen <ch...@apache.org> wrote:
>
> > https://github.com/apache/pulsar/issues/13340
> >
> > Pasted below for quoting convenience.
> >
> > ----
> > ### Motivation
> > The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> > https://github.com/apache/pulsar/pull/6772. It can resolve many load
> > balance issues of `OverloadShedder` and works well in many Pulsar
> > production clusters.
> >
> > In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> > policy is `OverloadShedder`.
> >
> > I think it's a good time for 2.10 to change default load balance
> > policy to `ThresholdShedder`, it will make throughput more balance
> > between brokers.
> >
> > ### Proposed Changes
> > In 2.10 release，for `broker.conf`, change
> > `loadBalancerLoadSheddingStrategy` from
> > `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> > `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
> >
>
Re: [DISCUSSION] PIP-122: Change loadBalancer default loadSheddingStrategy to ThresholdShedder
Posted by Sijie Guo <gu...@gmail.com>.
+1 ThresholdShedder is a much better policy
On Tue, Jan 4, 2022 at 6:24 AM Hang Chen <ch...@apache.org> wrote:
> https://github.com/apache/pulsar/issues/13340
>
> Pasted below for quoting convenience.
>
> ----
> ### Motivation
> The ThresholdShedder load balance policy since Pulsar 2.6.0 by
> https://github.com/apache/pulsar/pull/6772. It can resolve many load
> balance issues of `OverloadShedder` and works well in many Pulsar
> production clusters.
>
> In Pulsar 2.6.0, 2.7.0, 2.8.0 and 2.9.0, Pulsar's default load balance
> policy is `OverloadShedder`.
>
> I think it's a good time for 2.10 to change default load balance
> policy to `ThresholdShedder`, it will make throughput more balance
> between brokers.
>
> ### Proposed Changes
> In 2.10 release，for `broker.conf`, change
> `loadBalancerLoadSheddingStrategy` from
> `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder` to
> `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
>