You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/29 14:44:16 UTC
[GitHub] [pulsar] 315157973 opened a new pull request, #18675: [doc][broker]Add docs for ThresholdShedder
315157973 opened a new pull request, #18675:
URL: https://github.com/apache/pulsar/pull/18675
### Motivation
Add docs for ThresholdShedder
https://github.com/apache/pulsar/pull/17456
### Documentation
- [x] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 merged pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
315157973 merged PR #18675:
URL: https://github.com/apache/pulsar/pull/18675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#discussion_r1035465593
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
Review Comment:
```suggestion
However, in some special cases, the above default strategy cannot leverage the resources of low-load or idle machines.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#discussion_r1035469277
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be triggered, and there is one idle Broker with load of 0%.
+
+So, we add a new option `lowerBoundarySheddingEnabled`.
+If `lowerBoundarySheddingEnabled=true`, based on the `ThresholdShedder`, a leader will make the lower boundary judgment of the load.
Review Comment:
```suggestion
When `lowerBoundarySheddingEnabled` is set to `true`, a leader decides the lower boundary of the load.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#discussion_r1035456078
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be trigger, and there is one idle Broker with load of 0%.
+
+So, we add a new option `lowerBoundarySheddingEnabled`.
+if `lowerBoundarySheddingEnabled=true`, on the basis of `ThresholdShedder`, leader will make the lower boundary judgment of the load.
Review Comment:
```suggestion
If `lowerBoundarySheddingEnabled=true`, based on the `ThresholdShedder`, a leader will make the lower boundary judgment of the load.
```
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
Review Comment:
```suggestion
There are 11 brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
```
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be trigger, and there is one idle Broker with load of 0%.
+
+So, we add a new option `lowerBoundarySheddingEnabled`.
+if `lowerBoundarySheddingEnabled=true`, on the basis of `ThresholdShedder`, leader will make the lower boundary judgment of the load.
+When `current usage < average usage - threshold` (0% < 82.73% - 10%), the broker with the highest load will be triggered to unload.
Review Comment:
```suggestion
When `current usage < average usage - threshold` (e.g. 0% < 82.73% - 10%), the broker with the highest load will be triggered to unload.
```
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be trigger, and there is one idle Broker with load of 0%.
Review Comment:
```suggestion
Since 80% < 82.73%, unload will not be triggered, and there is one idle Broker with load of 0%.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#discussion_r1035465733
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 Brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be trigger, and there is one idle Broker with load of 0%.
+
+So, we add a new option `lowerBoundarySheddingEnabled`.
Review Comment:
```suggestion
To leverage the resources of low-load or idle machines, you can configure the `lowerBoundarySheddingEnabled` parameter on top of `ThresholdShedder`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#issuecomment-1331568871
@315157973 thanks for adding the docs!
Since it works on top of `ThresholdShedder`, does it make sense to move this newly added content a bit later ([starting from here](https://github.com/apache/pulsar/pull/18675/files#diff-28f860baebb586ef32c135e35e748de9be77675fe575018ce2e18d7f9405d660R168)), following the complete content of `ThresholdShedder`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#discussion_r1035469195
##########
site2/docs/administration-load-balance.md:
##########
@@ -132,6 +132,18 @@ This strategy tends to shed the bundles if any broker's usage is above the confi
For example, assume you have three brokers, the average broker usage of broker1 is 40%, the average broker usage of broker2 and broker3 is 10%, then the cluster average usage is 20% ((40% + 10% + 10%) / 3). If you set `loadBalancerBrokerThresholdShedderPercentage` to `10`, then only broker1's certain bundles get unloaded, because the average usage of broker1 is greater than the sum of the cluster average usage (20%) plus `loadBalancerBrokerThresholdShedderPercentage`(10%).
+However, in some special cases, the above default strategy may cause low-load machines to be idle forever.
+
+For example:
+
+There are 11 brokers, of which 10 are loaded at 80% and 1 is loaded at 0%.
+The average load is 80% * 10 / 11 = 72.73%, and the threshold to unload is 72.73% + 10% = 82.73%.
+Since 80% < 82.73%, unload will not be triggered, and there is one idle Broker with load of 0%.
+
+So, we add a new option `lowerBoundarySheddingEnabled`.
+If `lowerBoundarySheddingEnabled=true`, based on the `ThresholdShedder`, a leader will make the lower boundary judgment of the load.
+When `current usage < average usage - threshold` (e.g. 0% < 82.73% - 10%), the broker with the highest load will be triggered to unload.
Review Comment:
```suggestion
If `current usage` is less than `average usage - lower boundary load`, e.g., 0% < (82.73% - 10%), the broker with the highest load will be triggered to unload.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
315157973 commented on PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#issuecomment-1331730136
> @315157973 thanks for adding the docs! Since it works on top of `ThresholdShedder`, does it make sense to move this newly added content a bit later ([starting from Row168](https://github.com/apache/pulsar/pull/18675/files#diff-28f860baebb586ef32c135e35e748de9be77675fe575018ce2e18d7f9405d660R168)), following the complete content of `ThresholdShedder`?
This part explain the load balancing strategy. If we divide them into two parts, the meaning will be incoherent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #18675: [fix][doc]Add docs for ThresholdShedder
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #18675:
URL: https://github.com/apache/pulsar/pull/18675#issuecomment-1331735168
> > @315157973 thanks for adding the docs! Since it works on top of `ThresholdShedder`, does it make sense to move this newly added content a bit later ([starting from Row168](https://github.com/apache/pulsar/pull/18675/files#diff-28f860baebb586ef32c135e35e748de9be77675fe575018ce2e18d7f9405d660R168)), following the complete content of `ThresholdShedder`?
>
> This part explain the load balancing strategy. If we divide them into two parts, the meaning will be incoherent
Also make sense. My thought - it breaks the coherent content of `ThresholdShedder` and also has a dependency on the configuration of `ThresholdShedder`, so it looks a bit advancing if we introduce it ahead of `ThresholdShedder`. It's not a broker anyway :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org