You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/26 08:33:26 UTC
[GitHub] [pulsar] Nicklee007 opened a new pull request, #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Nicklee007 opened a new pull request, #16766:
URL: https://github.com/apache/pulsar/pull/16766
### Motivation
As the `brokerAvgResourceUsageWithWeight` stored the broker's history resource usage, if a broker is high load before restart, after the broker restart always show a high load history resource usage, and `loadBalancerHistoryResourcePercentage = 0.9`, which will cause the first few times the broker can not be select as best broker to load bundles. Only when `selectBrokerForAssignment` invoke few time's , the `brokerAvgResourceUsageWithWeight` could be neutralized by the current resourceUsage.
But bundles unload is not a frequently operator if not disable `doLoadShedding`,it will wait a long time; So if a broker restarted we can clean the brokers history resource usage to guaranty the no bundles or no msgRate broker can be select as the one of best brokers.
https://github.com/apache/pulsar/blob/bc94643bc1a7f365dfb75e389ac3e1156770a119/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java#L99-L100
### Modifications
1. If the broker not own bundles or the msgRate is 0, use current resourceUsage to cover the historyUsage.
2. add some unit test.
### Documentation
- [X] `doc-not-needed`
(Please explain why)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1202062256
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1287584515
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1228189484
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1342273658
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/16766?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#16766](https://codecov.io/gh/apache/pulsar/pull/16766?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3c0c603) into [master](https://codecov.io/gh/apache/pulsar/commit/99e26f5e943e3978bba135a2c17ff08dc9b9599a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (99e26f5) will **decrease** coverage by `7.76%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/pulsar/pull/16766?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #16766 +/- ##
============================================
- Coverage 44.79% 37.03% -7.77%
+ Complexity 10026 1966 -8060
============================================
Files 703 209 -494
Lines 68808 14425 -54383
Branches 7375 1574 -5801
============================================
- Hits 30822 5342 -25480
+ Misses 34378 8496 -25882
+ Partials 3608 587 -3021
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `37.03% <ø> (-7.77%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/16766?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pulsar/client/impl/PulsarClientImpl.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1B1bHNhckNsaWVudEltcGwuamF2YQ==) | `43.85% <0.00%> (-0.95%)` | :arrow_down: |
| [...apache/pulsar/client/impl/AutoClusterFailover.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0F1dG9DbHVzdGVyRmFpbG92ZXIuamF2YQ==) | `69.44% <0.00%> (-0.56%)` | :arrow_down: |
| [...he/pulsar/client/impl/MultiTopicsConsumerImpl.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL011bHRpVG9waWNzQ29uc3VtZXJJbXBsLmphdmE=) | `22.86% <0.00%> (-0.12%)` | :arrow_down: |
| [...loadbalance/impl/LeastResourceUsageWithWeight.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9sb2FkYmFsYW5jZS9pbXBsL0xlYXN0UmVzb3VyY2VVc2FnZVdpdGhXZWlnaHQuamF2YQ==) | | |
| [...apache/bookkeeper/mledger/ManagedLedgerConfig.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9NYW5hZ2VkTGVkZ2VyQ29uZmlnLmphdmE=) | | |
| [...roker/service/PulsarMetadataEventSynchronizer.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1B1bHNhck1ldGFkYXRhRXZlbnRTeW5jaHJvbml6ZXIuamF2YQ==) | | |
| [...he/bookkeeper/mledger/offload/OffloadersCache.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9vZmZsb2FkL09mZmxvYWRlcnNDYWNoZS5qYXZh) | | |
| [...ker/service/persistent/PersistentSubscription.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvUGVyc2lzdGVudFN1YnNjcmlwdGlvbi5qYXZh) | | |
| [...er/transaction/exception/TransactionException.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9leGNlcHRpb24vVHJhbnNhY3Rpb25FeGNlcHRpb24uamF2YQ==) | | |
| [...ervice/AbstractDispatcherSingleActiveConsumer.java](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0Fic3RyYWN0RGlzcGF0Y2hlclNpbmdsZUFjdGl2ZUNvbnN1bWVyLmphdmE=) | | |
| ... and [487 more](https://codecov.io/gh/apache/pulsar/pull/16766/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1341888243
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin merged pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
gaozhangmin merged PR #16766:
URL: https://github.com/apache/pulsar/pull/16766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1193963803
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 closed pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 closed pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
URL: https://github.com/apache/pulsar/pull/16766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1194069741
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1226854139
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Nicklee007 commented on pull request #16766: [improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #16766:
URL: https://github.com/apache/pulsar/pull/16766#issuecomment-1231849018
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org