You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "heesung-sn (via GitHub)" <gi...@apache.org> on 2023/03/15 01:37:22 UTC
[GitHub] [pulsar] heesung-sn opened a new pull request, #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
heesung-sn opened a new pull request, #19813:
URL: https://github.com/apache/pulsar/pull/19813
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->
<!-- or this PR is one task of an issue -->
Master Issue: Master Issue: https://github.com/apache/pulsar/issues/16691, https://github.com/apache/pulsar/issues/18099
### Motivation
Raising a PR to implement Master Issue: https://github.com/apache/pulsar/issues/16691, https://github.com/apache/pulsar/issues/18099
We want to reduce unload frequencies from flaky traffic.
### Modifications
This PR
- Introduced a config `loadBalancerSheddingConditionHitCountThreshold` to further restrict shedding conditions based on the hit count.
- Lowered the default `loadBalanceSheddingDelayInSeconds` value from 600 to 180, as 10 mins are too long. 3 mins can be long enough to catch the new load after unloads.
- Changed the config `loadBalancerBundleLoadReportPercentage` to `loadBalancerMaxNumberOfBundlesInBundleLoadReport` to make the topk bundle count absolute instead of relative.
- Renamed `loadBalancerNamespaceBundleSplitConditionThreshold` to `loadBalancerNamespaceBundleSplitConditionHitCountThreshold` to be consistent with `*ConditionHitCountThreshold`.
- Renamed `loadBalancerMaxNumberOfBrokerTransfersPerCycle ` to `loadBalancerMaxNumberOfBrokerSheddingPerCycle`.
- Added LoadDataStore cleanup logic in BSC monitor.
- Added `msgThroughputEMA` in BrokerLoadData to smooth the broker throughput info.
- Updated Topk bundles sorted in a ascending order (instead of descending)
- Update some info logs to only show in the debug mode.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- *Updated unit tests.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
We will have separate PRs to update the Doc later.
### Matching PR in forked repository
PR in forked repository: https://github.com/heesung-sn/pulsar/pull/39
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
URL: https://github.com/apache/pulsar/pull/19813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1144124111
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -567,8 +616,7 @@ private void setIsolationPolicies(SimpleResourceAllocationPolicies policies,
}).when(policies).shouldFailoverToSecondaries(eq(namespaceName), anyInt());
}
- private PulsarService getMockPulsar() {
- var pulsar = mock(PulsarService.class);
+ private PulsarService getMockPulsar(PulsarService pulsar) {
Review Comment:
updated.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -589,8 +637,9 @@ private PulsarService getMockPulsar() {
@Test
public void testBundlesWithAntiAffinityGroup() throws IllegalAccessException, MetadataStoreException {
- var pulsar = getMockPulsar();
- TransferShedder transferShedder = new TransferShedder(pulsar, antiAffinityGroupPolicyHelper);
+ //var pulsar = getMockPulsar();
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1475105049
Yes, I will rebase after the pr for excluding bundles with policies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "gaoran10 (via GitHub)" <gi...@apache.org>.
gaoran10 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1147078266
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -1057,7 +1079,12 @@ private void doCleanup(String broker) throws ExecutionException, InterruptedExce
double cleanupTime = TimeUnit.NANOSECONDS
.toMillis((System.nanoTime() - startTime));
- // TODO: clean load data stores
+
+ // clean load data stores
+ getContext().topBundleLoadDataStore().removeAsync(broker);
+ getContext().brokerLoadDataStore().removeAsync(broker);
+
+ cleanupJobs.remove(broker);
Review Comment:
Do we need to cancel the completable future?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
Review Comment:
Do we need to add the error param?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -156,34 +178,42 @@ void offload(double max, double min, double offload) {
double maxd = Math.max(0, max - offload);
double mind = min + offload;
sqSum += maxd * maxd + mind * mind;
- std = Math.sqrt(sqSum / totalBrokers - avg * avg);
+ std = Math.sqrt(Math.abs(sqSum / totalBrokers - avg * avg));
Review Comment:
Does the param `std` base on this formula?
<img width="250" alt="image" src="https://user-images.githubusercontent.com/15029908/227688159-c1e19ea2-cead-46f1-baa0-8f6853d28424.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1144124188
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -134,25 +143,29 @@ public void init() throws MetadataStoreException {
(upperEndpoint.equals(NamespaceBundles.FULL_UPPER_BOUND)) ? BoundType.CLOSED : BoundType.OPEN);
return new NamespaceBundle(NamespaceName.get(namespace), hashRange, factory);
}).when(factory).getBundle(anyString(), anyString());
- doReturn(true).when(antiAffinityGroupPolicyHelper).canUnload(any(), any(), any(), any());
+ doReturn(new AtomicReference(loadManagerWrapper)).when(pulsar).getLoadManager();
Review Comment:
updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -195,47 +209,65 @@ Optional<UnloadDecision.Reason> update(final LoadDataStore<BrokerLoadData> loadS
double load = localBrokerData.getWeightedMaxEMA();
- minBrokers.offer(broker);
- if (minBrokers.size() > maxTransfers) {
- minBrokers.poll();
- }
- maxBrokers.offer(broker);
- if (maxBrokers.size() > maxTransfers) {
- maxBrokers.poll();
- }
sum += load;
sqSum += load * load;
totalBrokers++;
}
-
if (totalBrokers == 0) {
if (decisionReason == null) {
decisionReason = NoBrokers;
}
update(0.0, 0.0, 0);
+ if (debug) {
+ log.info("There is no broker load data.");
+ }
+ return Optional.of(decisionReason);
+ }
+
+ if (!missingLoadDataBrokers.isEmpty()) {
+ decisionReason = NoLoadData;
+ update(0.0, 0.0, 0);
+ if (debug) {
+ log.info("There is missing load data from brokers:{}", missingLoadDataBrokers);
+ }
return Optional.of(decisionReason);
}
update(sum, sqSum, totalBrokers);
return Optional.empty();
}
- boolean hasTransferableBrokers() {
- return !(maxBrokers.isEmpty() || minBrokers.isEmpty()
- || maxBrokers.peekLast().equals(minBrokers().peekLast()));
- }
-
void setLoadDataStore(LoadDataStore<BrokerLoadData> loadDataStore) {
this.loadDataStore = loadDataStore;
+ brokersSortedByLoad.addAll(loadDataStore.entrySet());
+ Collections.sort(brokersSortedByLoad, (a, b) -> Double.compare(
+ a.getValue().getWeightedMaxEMA(),
+ b.getValue().getWeightedMaxEMA()));
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1146755749
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
+ lastTombstonedAt = lastSuccessfulTombstonedAt;
+ } else {
+ boolean debug = ExtensibleLoadManagerImpl.debug(conf, log);
+ if (debug) {
+ log.info("Cleaned broker load data.");
+ }
+ }
+ }
+ );
+
+ }
+
+ @Override
+ public void handleEvent(String serviceUnit, ServiceUnitStateData data, Throwable t) {
+ executor.execute(() -> {
Review Comment:
The intention was to control this `handleEvents` in a single-threaded manner. I agree that we don't have to use executor here. Removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1146782748
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -122,17 +148,14 @@ static class LoadStats {
private int totalBrokers;
private double avg;
private double std;
- private MinMaxPriorityQueue<String> minBrokers;
- private MinMaxPriorityQueue<String> maxBrokers;
private LoadDataStore<BrokerLoadData> loadDataStore;
-
+ private List<Map.Entry<String, BrokerLoadData>> brokersSortedByLoad;
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1146756985
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/SplitScheduler.java:
##########
@@ -156,6 +159,13 @@ public void start() {
task = loadManagerExecutor.scheduleAtFixedRate(() -> {
try {
execute();
+ if (conf.isLoadBalancerDebugModeEnabled()) {
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1474573895
> Please pause the review. There will be more updates.
Please continue the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1142829624
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/models/TopKBundles.java:
##########
@@ -66,7 +66,7 @@ public void update(Map<String, NamespaceBundleStats> bundleStats, int topk) {
topk = Math.min(topk, arr.size());
partitionSort(arr, topk);
- for (int i = 0; i < topk; i++) {
+ for (int i = topk - 1; i >= 0; i--) {
Review Comment:
We want to transfer lower-loaded ones first to meet the target threshold. If the highest is too much traffic, we can't unload at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148429116
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -156,34 +178,42 @@ void offload(double max, double min, double offload) {
double maxd = Math.max(0, max - offload);
double mind = min + offload;
sqSum += maxd * maxd + mind * mind;
- std = Math.sqrt(sqSum / totalBrokers - avg * avg);
+ std = Math.sqrt(Math.abs(sqSum / totalBrokers - avg * avg));
Review Comment:
yes, and more specifically,
<img width="597" alt="Screen Shot 2023-03-25 at 1 31 55 PM" src="https://user-images.githubusercontent.com/103456639/227740224-c2c1ce6a-9c62-4698-90e6-0829f36ecef0.png">
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -1057,7 +1079,12 @@ private void doCleanup(String broker) throws ExecutionException, InterruptedExce
double cleanupTime = TimeUnit.NANOSECONDS
.toMillis((System.nanoTime() - startTime));
- // TODO: clean load data stores
+
+ // clean load data stores
+ getContext().topBundleLoadDataStore().removeAsync(broker);
+ getContext().brokerLoadDataStore().removeAsync(broker);
+
+ cleanupJobs.remove(broker);
Review Comment:
no. it will be completed after this call returns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1145759608
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
+ lastTombstonedAt = lastSuccessfulTombstonedAt;
+ } else {
+ boolean debug = ExtensibleLoadManagerImpl.debug(conf, log);
+ if (debug) {
+ log.info("Cleaned broker load data.");
+ }
+ }
+ }
+ );
+
+ }
+
+ @Override
+ public void handleEvent(String serviceUnit, ServiceUnitStateData data, Throwable t) {
+ executor.execute(() -> {
Review Comment:
Is there any reason to use executor here? I did not see any block operating here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1145760495
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/SplitScheduler.java:
##########
@@ -156,6 +159,13 @@ public void start() {
task = loadManagerExecutor.scheduleAtFixedRate(() -> {
try {
execute();
+ if (conf.isLoadBalancerDebugModeEnabled()) {
Review Comment:
Why don't we use `debugMode` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1146027666
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -122,17 +148,14 @@ static class LoadStats {
private int totalBrokers;
private double avg;
private double std;
- private MinMaxPriorityQueue<String> minBrokers;
- private MinMaxPriorityQueue<String> maxBrokers;
private LoadDataStore<BrokerLoadData> loadDataStore;
-
+ private List<Map.Entry<String, BrokerLoadData>> brokersSortedByLoad;
Review Comment:
```suggestion
private List<Map.Entry<String, BrokerLoadData>> brokersSortedByLoad;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1475093352
Please resolve the conflicts and rebase it into the master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148430277
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -1057,7 +1079,12 @@ private void doCleanup(String broker) throws ExecutionException, InterruptedExce
double cleanupTime = TimeUnit.NANOSECONDS
.toMillis((System.nanoTime() - startTime));
- // TODO: clean load data stores
+
+ // clean load data stores
+ getContext().topBundleLoadDataStore().removeAsync(broker);
+ getContext().brokerLoadDataStore().removeAsync(broker);
+
+ cleanupJobs.remove(broker);
Review Comment:
No, it will complete when this func returns. In fact, I think we better move this `cleanupJobs.remove(broker);` in the finally block after this call. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148430365
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -156,34 +178,42 @@ void offload(double max, double min, double offload) {
double maxd = Math.max(0, max - offload);
double mind = min + offload;
sqSum += maxd * maxd + mind * mind;
- std = Math.sqrt(sqSum / totalBrokers - avg * avg);
+ std = Math.sqrt(Math.abs(sqSum / totalBrokers - avg * avg));
Review Comment:
Yes, (but the derived version)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1486216393
@Demogorgon314 can we merge this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148428453
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
Review Comment:
yes. updated.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -156,34 +178,42 @@ void offload(double max, double min, double offload) {
double maxd = Math.max(0, max - offload);
double mind = min + offload;
sqSum += maxd * maxd + mind * mind;
- std = Math.sqrt(sqSum / totalBrokers - avg * avg);
+ std = Math.sqrt(Math.abs(sqSum / totalBrokers - avg * avg));
Review Comment:
yes, and more specifically,
<img width="597" alt="Screen Shot 2023-03-25 at 1 31 55 PM" src="https://user-images.githubusercontent.com/103456639/227740224-c2c1ce6a-9c62-4698-90e6-0829f36ecef0.png">
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java:
##########
@@ -1057,7 +1079,12 @@ private void doCleanup(String broker) throws ExecutionException, InterruptedExce
double cleanupTime = TimeUnit.NANOSECONDS
.toMillis((System.nanoTime() - startTime));
- // TODO: clean load data stores
+
+ // clean load data stores
+ getContext().topBundleLoadDataStore().removeAsync(broker);
+ getContext().brokerLoadDataStore().removeAsync(broker);
+
+ cleanupJobs.remove(broker);
Review Comment:
no. it will be completed after this call returns.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
Review Comment:
yes, added. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148428453
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/reporter/BrokerLoadDataReporter.java:
##########
@@ -152,4 +180,49 @@ protected double percentChange(final double oldValue, final double newValue) {
}
return 100 * Math.abs((oldValue - newValue) / oldValue);
}
+
+ @VisibleForTesting
+ protected void tombstone() {
+ var now = System.currentTimeMillis();
+ if (now - lastTombstonedAt < tombstoneDelayInMillis) {
+ return;
+ }
+ var lastSuccessfulTombstonedAt = lastTombstonedAt;
+ lastTombstonedAt = now; // dedup first
+ brokerLoadDataStore.removeAsync(lookupServiceAddress)
+ .whenComplete((__, e) -> {
+ if (e != null) {
+ log.error("Failed to clean broker load data.");
Review Comment:
yes. updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "gaoran10 (via GitHub)" <gi...@apache.org>.
gaoran10 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1148475972
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -156,34 +178,42 @@ void offload(double max, double min, double offload) {
double maxd = Math.max(0, max - offload);
double mind = min + offload;
sqSum += maxd * maxd + mind * mind;
- std = Math.sqrt(sqSum / totalBrokers - avg * avg);
+ std = Math.sqrt(Math.abs(sqSum / totalBrokers - avg * avg));
Review Comment:
Ok, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 merged pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 merged PR #19813:
URL: https://github.com/apache/pulsar/pull/19813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1478776148
Rebase is done. Please continue the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Demogorgon314 commented on a diff in pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "Demogorgon314 (via GitHub)" <gi...@apache.org>.
Demogorgon314 commented on code in PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#discussion_r1141831538
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -134,25 +143,29 @@ public void init() throws MetadataStoreException {
(upperEndpoint.equals(NamespaceBundles.FULL_UPPER_BOUND)) ? BoundType.CLOSED : BoundType.OPEN);
return new NamespaceBundle(NamespaceName.get(namespace), hashRange, factory);
}).when(factory).getBundle(anyString(), anyString());
- doReturn(true).when(antiAffinityGroupPolicyHelper).canUnload(any(), any(), any(), any());
+ doReturn(new AtomicReference(loadManagerWrapper)).when(pulsar).getLoadManager();
Review Comment:
```suggestion
doReturn(new AtomicReference<>(loadManagerWrapper)).when(pulsar).getLoadManager();
```
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -567,8 +616,7 @@ private void setIsolationPolicies(SimpleResourceAllocationPolicies policies,
}).when(policies).shouldFailoverToSecondaries(eq(namespaceName), anyInt());
}
- private PulsarService getMockPulsar() {
- var pulsar = mock(PulsarService.class);
+ private PulsarService getMockPulsar(PulsarService pulsar) {
Review Comment:
This method is never used right now. Please remove it.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedder.java:
##########
@@ -195,47 +209,65 @@ Optional<UnloadDecision.Reason> update(final LoadDataStore<BrokerLoadData> loadS
double load = localBrokerData.getWeightedMaxEMA();
- minBrokers.offer(broker);
- if (minBrokers.size() > maxTransfers) {
- minBrokers.poll();
- }
- maxBrokers.offer(broker);
- if (maxBrokers.size() > maxTransfers) {
- maxBrokers.poll();
- }
sum += load;
sqSum += load * load;
totalBrokers++;
}
-
if (totalBrokers == 0) {
if (decisionReason == null) {
decisionReason = NoBrokers;
}
update(0.0, 0.0, 0);
+ if (debug) {
+ log.info("There is no broker load data.");
+ }
+ return Optional.of(decisionReason);
+ }
+
+ if (!missingLoadDataBrokers.isEmpty()) {
+ decisionReason = NoLoadData;
+ update(0.0, 0.0, 0);
+ if (debug) {
+ log.info("There is missing load data from brokers:{}", missingLoadDataBrokers);
+ }
return Optional.of(decisionReason);
}
update(sum, sqSum, totalBrokers);
return Optional.empty();
}
- boolean hasTransferableBrokers() {
- return !(maxBrokers.isEmpty() || minBrokers.isEmpty()
- || maxBrokers.peekLast().equals(minBrokers().peekLast()));
- }
-
void setLoadDataStore(LoadDataStore<BrokerLoadData> loadDataStore) {
this.loadDataStore = loadDataStore;
+ brokersSortedByLoad.addAll(loadDataStore.entrySet());
+ Collections.sort(brokersSortedByLoad, (a, b) -> Double.compare(
+ a.getValue().getWeightedMaxEMA(),
+ b.getValue().getWeightedMaxEMA()));
Review Comment:
nit:
```suggestion
brokersSortedByLoad.sort(Comparator.comparingDouble(a -> a.getValue().getWeightedMaxEMA()));
```
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/scheduler/TransferShedderTest.java:
##########
@@ -589,8 +637,9 @@ private PulsarService getMockPulsar() {
@Test
public void testBundlesWithAntiAffinityGroup() throws IllegalAccessException, MetadataStoreException {
- var pulsar = getMockPulsar();
- TransferShedder transferShedder = new TransferShedder(pulsar, antiAffinityGroupPolicyHelper);
+ //var pulsar = getMockPulsar();
Review Comment:
We can remove it.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/models/TopKBundles.java:
##########
@@ -66,7 +66,7 @@ public void update(Map<String, NamespaceBundleStats> bundleStats, int topk) {
topk = Math.min(topk, arr.size());
partitionSort(arr, topk);
- for (int i = 0; i < topk; i++) {
+ for (int i = topk - 1; i >= 0; i--) {
Review Comment:
Why reverse the topK bundles order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on pull request #19813: [improve][broker] PIP-192 Improved Auto Unload Logic
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on PR #19813:
URL: https://github.com/apache/pulsar/pull/19813#issuecomment-1471317765
Please pause the review. There will be more updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org