You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/03 11:43:44 UTC
[GitHub] [pulsar] suiyuzeng opened a new issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
suiyuzeng opened a new issue #13119:
URL: https://github.com/apache/pulsar/issues/13119
**Describe the bug**
Upgrading a cluster which has 3 nodes, the last node will not be assigned a bundle for a long time even if the load of other 2 nodes is very high.
```
private void updateAvgResourceUsage(String broker, LocalBrokerData localBrokerData, final double historyPercentage,
final ServiceConfiguration conf) {
double historyUsage =
brokerAvgResourceUsage.getOrDefault(broker, 0.0);
historyUsage = historyUsage * historyPercentage
+ (1 - historyPercentage) * localBrokerData.getMaxResourceUsageWithWeight(
conf.getLoadBalancerCPUResourceWeight(),
conf.getLoadBalancerMemoryResourceWeight(), conf.getLoadBalancerDirectMemoryResourceWeight(),
conf.getLoadBalancerBandwithInResourceWeight(),
conf.getLoadBalancerBandwithOutResourceWeight());
brokerAvgResourceUsage.put(broker, historyUsage);
}
```
In org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder#updateAvgResourceUsage, if a broker change to be a leader, brokerAvgResourceUsage has no recoder about any broker. As the historyUsage was set to be 0, it wil take a long time to be actual usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-986040175
> @hangc0276 Maybe we can make each broker calculate it's own historyUsage, and upload to zk like TimeAverageBrokerData ?
@Jason918 Considering not import useless fields for other shedder strategy, we can use the current resource usage value as history usage value when leader change to speed up getting the actual value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-985957888
@hangc0276 Maybe we can make each broker calculate it's own historyUsage, and upload to zk like TimeAverageBrokerData ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-986164656
@suiyuzeng @hangc0276 I met this problem too.
One way to reduce the time for load balance is to set the value of `loadBalancerHistoryResourcePercentage` to a lower value, for example, 0.5 instream of the default 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] suiyuzeng commented on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
suiyuzeng commented on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-988778352
> @suiyuzeng @hangc0276 I met this problem too. One way to reduce the time for load balance is to set the value of `loadBalancerHistoryResourcePercentage` to a lower value, for example, 0.5 instread of the default 0.9.
It will work for this issue. But some time, we may need a high percentage of the history.
I think the threshold strategy should recorder the history even if the leader change. How about this way:
1. store the history in zk.
2. if a broker restart, discard the history. If the czxid of the broker in ledger/available change, discard the history.
@hangc0276 @Jason918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-986164656
@suiyuzeng @hangc0276 I met this problem too.
One way to reduce the time for load balance is to set the value of `loadBalancerHistoryResourcePercentage` to a lower value, for example, 0.5 instread of the default 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #13119:
URL: https://github.com/apache/pulsar/issues/13119#issuecomment-986164656
@suiyuzeng @hangc0276 I met this problem too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #13119:
URL: https://github.com/apache/pulsar/issues/13119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #13119: Load balance shedding not in time with the strategy of ThresholdShedder
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #13119:
URL: https://github.com/apache/pulsar/issues/13119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org