You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "gaozhangmin (via GitHub)" <gi...@apache.org> on 2025/07/30 03:02:15 UTC
[PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
gaozhangmin opened a new pull request, #24581:
URL: https://github.com/apache/pulsar/pull/24581
# Background
In the current implementation, AvgShedder relies on bundleBrokerMap to make broker selection decisions.
However, this map is empty before load shedding occurs, causing selectBroker() to fall back to random broker assignment.
When users disable load shedding, this results in all bundles being assigned randomly, leading to unbalanced load distribution across brokers, which is undesirable and inconsistent with production expectations.
# Enhancement
Prior to the introduction of AvgShedder, the default broker selection strategy was LeastLongTermMessageRate.
To address the issue of randomness and ensure a more reasonable broker assignment:
AvgShedder is now modified to fallback to LeastLongTermMessageRate
when bundleBrokerMap is empty during broker selection.
# Benefit
Ensures stable and deterministic broker assignment, even if load shedding is disabled
Prevents random distribution of bundles, improving cluster balance and consistency
Maintains backward compatibility with the previously used strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "gaozhangmin (via GitHub)" <gi...@apache.org>.
gaozhangmin closed pull request #24581: alternative to LeastLongTermMessageRate rather than select broker ran…
URL: https://github.com/apache/pulsar/pull/24581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#issuecomment-3134710362
@gaozhangmin Please add the following content to your PR description and select a checkbox:
```
- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#issuecomment-3135329141
Have you performed enough performance test on this PR to verify that it offers improvements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#issuecomment-3135350268
I don't agree with removing the old logic. `AvgShedder` should not interfere with other load balancing algorithms, as the existing algorithms have complex relationships that would create chaos.
By the way, it isn't random selection logic; it's hash-based logic, which distributes bundles evenly across all brokers, as opposed to randomness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#discussion_r2241427792
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/AvgShedder.java:
##########
@@ -285,9 +288,7 @@ public Optional<String> selectBroker(Set<String> candidates, BundleData bundleTo
candidates);
}
}
- String broker = getExpectedBroker(candidates, bundleToAssign);
- bundleBrokerMap.put(bundleToAssign, broker);
- return Optional.of(broker);
+ return leastLongTermMessageRatePlacement.selectBroker(candidates, bundleToAssign, loadData, conf);
Review Comment:
The removal of bundleBrokerMap.put(bundleToAssign, broker) may cause issues if other parts of the code expect this map to be populated. Consider whether the selected broker should still be cached in bundleBrokerMap for consistency with the rest of the class logic.
```suggestion
Optional<String> selectedBroker = leastLongTermMessageRatePlacement.selectBroker(candidates, bundleToAssign, loadData, conf);
selectedBroker.ifPresent(broker -> bundleBrokerMap.put(bundleToAssign, broker));
return selectedBroker;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#issuecomment-3135216867
@gaozhangmin Please set a proper title for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] alternative to LeastLongTermMessageRate rather than select broker ran… [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24581:
URL: https://github.com/apache/pulsar/pull/24581#issuecomment-3135213374
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org