You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "hanmz (via GitHub)" <gi...@apache.org> on 2024/01/16 05:24:34 UTC
[PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnload… [pulsar]
hanmz opened a new pull request, #21906:
URL: https://github.com/apache/pulsar/pull/21906
Fixes https://github.com/apache/pulsar/issues/21905
### Motivation
In class UniformLoadShedder, we add a fixed delta value of 1000 when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles.This will cause the load balancing strategy to fail in low traffic scenarios.
### Modifications
Remove `delta` when calculating `msgRateRequiredFromUnloadedBundles` and `msgThroughputRequiredFromUnloadedBundles`
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#issuecomment-1893542333
could we add a test for this case ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1477439085
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
I see. Plz update the motivation in the description that this threshold(delta) can be controlled by minUnloadMessage and minUnloadMessageThroughput. Hence , keeping this delta is redundant or error prone for the smaller traffic too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1468036684
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
what was the original intention for this 1000 delta? If removing this, would that cause any negative impact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1477196573
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
I understand that delta=1000 here does not have much meaning. The initial design was just to allow the data of one shedding to reach a certain threshold and to prevent the data flow of one shedding from being too small.
However, later we added the configuration minUnloadMessage and minUnloadMessageThroughput. These two configurations can completely replace the role of delta=1000, so the delta here is meaningless.
However, due to the existence of delta=1000, broker load balancing cannot be achieved in small traffic scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1477639088
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
OK, I have updated motivation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#issuecomment-1925618654
Please help me find out what is the purpose of adding delta=1000 here? @lordcheng10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1477439085
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
I see. Plz update the motivation in the description that this threshold(delta) can be controlled by minUnloadMessage and minUnloadMessageThroughput. Hence , keeping this delta is redundant and error prone for the smaller traffic too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "Jason918 (via GitHub)" <gi...@apache.org>.
Jason918 merged PR #21906:
URL: https://github.com/apache/pulsar/pull/21906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#issuecomment-2029275608
> @hanmz Could you merge the master branch, and then the CI is passed, I will merge this PR.
Okay, I'll do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#issuecomment-2006459591
@hanmz Could you merge the master branch, and then the CI is passed, I will merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#discussion_r1468037959
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/UniformLoadShedder.java:
##########
@@ -154,8 +154,7 @@ public Multimap<String, String> findBundlesForUnloading(final LoadData loadData,
}
String bundle = e.getLeft();
double bundleMsgRate = e.getRight();
- if (bundleMsgRate <= (msgRateRequiredFromUnloadedBundles.getValue()
- + 1000/* delta */)) {
Review Comment:
Can we use a multiplication delta, like 1%, to keep this delta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Remove delta when calculating msgRateRequiredFromUnloadedBundles and msgThroughputRequiredFromUnloadedBundles for UniformLoadShedder [pulsar]
Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on PR #21906:
URL: https://github.com/apache/pulsar/pull/21906#issuecomment-1894867285
> could we add a test for this case ?
Okay, I'll add a test case here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org