You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/14 15:10:40 UTC
[GitHub] [pulsar] HQebupt opened a new pull request, #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
HQebupt opened a new pull request, #16059:
URL: https://github.com/apache/pulsar/pull/16059
### Motivation
When we use `ModualLoadManager` as `LoadManager` in the pulsar cluster, and the load balancer shedding strategy is `ThresholdShedder`, we found that the unloaded bundles might be loaded by another broker which has resource usage above average load, and make the new broker be overloaded again, which frequently causes unloading bundles.
```
loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerBrokerThresholdShedderPercentage=10
```
We found that it tooks more than 10 hours and 400+ times to doing load shedding in the cluster. And the cluster continue to unload bundles when some of topics are lots of traffic in a short time. Below is the unload metric.
<img width="866" alt="image" src="https://user-images.githubusercontent.com/4970972/173602319-95e7b7a3-e2c9-45c3-a9af-9cd7e88ea227.png">
The key reason is that some brokers with lower load but more bundles can not be candidate due to distributing bundles evenly in LoadManager by force.
<img width="1186" alt="image" src="https://user-images.githubusercontent.com/4970972/173603516-2efefae4-d22b-445c-9414-6744d22e0323.png">
It could be much better to disable distribute bundles evenly in `LoadManager`, which can select the broker from those having resource usage below average load, so it can prevent the least loaded broker from quickly becoming heavily loaded.
Therefore, it recommend that enable distribute bundles evenly among all brokers by customers according to user scenarios .
After disable distribute bundles evenly in `LoadManager`, the brokers with lower load but more bunldes can be candidate. It reduce the unload times and the cluster is stable with even load on each broker as follows.
<img width="1661" alt="image" src="https://user-images.githubusercontent.com/4970972/173606832-92eeac3f-89ff-4291-b90d-2a6de7ad3b84.png">
### Modifications
- Add a field property `loadBalancerDistributeBundlesEvenlyEnabled` for Load Balancer in `ServiceConfiguration`, keep it to be true as default.
- Add a switch to enable distribute bundles evenly in `ModularLoadManager`
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change is a trivial rework / code cleanup without any test coverage.
### Does this pull request potentially affect one of the following parts:
If `yes` was chosen, please highlight the changes
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
- [x] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 merged pull request #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
Posted by GitBox <gi...@apache.org>.
Jason918 merged PR #16059:
URL: https://github.com/apache/pulsar/pull/16059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #16059:
URL: https://github.com/apache/pulsar/pull/16059#issuecomment-1163853436
@hangc0276 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #16059:
URL: https://github.com/apache/pulsar/pull/16059#issuecomment-1615342222
Discussing 2.10 cherry-pick here: https://lists.apache.org/thread/2mq3h9gpqv1b4zyyp2cddfltlqz3wtg0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] HQebupt commented on pull request #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
Posted by GitBox <gi...@apache.org>.
HQebupt commented on PR #16059:
URL: https://github.com/apache/pulsar/pull/16059#issuecomment-1156343077
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #16059: [improve][broker] add switch for enable/disable distribute bundles evenly in LoadManager
Posted by GitBox <gi...@apache.org>.
Jason918 commented on code in PR #16059:
URL: https://github.com/apache/pulsar/pull/16059#discussion_r897459511
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##########
@@ -1990,6 +1990,14 @@ public class ServiceConfiguration implements PulsarConfiguration {
doc = "Load shedding interval. \n\nBroker periodically checks whether some traffic"
+ " should be offload from some over-loaded broker to other under-loaded brokers"
)
+
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "enable/disable distribute bundles evenly"
+ )
+ private boolean loadBalancerDistributeBundlesEvenlyEnabled = true;
Review Comment:
This should be added after L2001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org