You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "AuroraTwinkle (via GitHub)" <gi...@apache.org> on 2024/06/14 16:37:30 UTC
[I] [Bug] Pulsar broker CPU stratification problem [pulsar]
AuroraTwinkle opened a new issue, #22911:
URL: https://github.com/apache/pulsar/issues/22911
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
**JDK:**
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)
**OS:**
Linux pulsarbrokerk8s-public01-100 5.4.119-19-0009_plus #1 SMP Wed Dec 29 13:51:43 CST 2021 x86_64 GNU/Linux
**Pulsar**:
3.0.3
### Minimal reproduce step
1. The automatic load balancing feature is turned on, and a large amount of traffic is written, 30W+ TPS
2. The configs for loadBalancer are like:
```
loadBalancerEnabled=true
loadBalancerReportUpdateThresholdPercentage=10
loadBalancerReportUpdateMinIntervalMillis=5000
loadBalancerReportUpdateMaxIntervalMinutes=15
loadBalancerHostUsageCheckIntervalMinutes=1
loadBalancerSheddingEnabled=true
loadBalancerSheddingIntervalMinutes=1
loadBalancerSheddingGracePeriodMinutes=30
loadBalancerBrokerMaxTopics=50000
loadBalancerBrokerOverloadedThresholdPercentage=85
loadBalancerResourceQuotaUpdateIntervalMinutes=15
loadBalancerAutoBundleSplitEnabled=true
loadBalancerAutoUnloadSplitBundlesEnabled=true
loadBalancerNamespaceBundleMaxTopics=1000
loadBalancerNamespaceBundleMaxSessions=1000
loadBalancerNamespaceBundleMaxMsgRate=30000
loadBalancerNamespaceBundleMaxBandwidthMbytes=100
loadBalancerNamespaceMaximumBundles=128
loadBalancerOverrideBrokerNicSpeedGbps=1
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastResourceUsageWithWeight
loadBalancerBrokerThresholdShedderPercentage=5
loadBalancerAverageResourceUsageDifferenceThresholdPercentage=5
loadBalancerMsgRateDifferenceShedderThreshold=50
loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=4
loadBalancerHistoryResourcePercentage=0.5
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=1.0
loadBalancerDirectMemoryResourceWeight=1.0
loadBalancerBundleUnloadMinThroughputThreshold=10
loadBalancerDebugModeEnabled=false
loadBalancerBrokerLoadTargetStd=0.25
loadBalancerSheddingConditionHitCountThreshold=3
loadBalancerTransferEnabled=true
loadBalancerMaxNumberOfBrokerSheddingPerCycle=3
loadBalancerBrokerLoadDataTTLInSeconds=1800
loadBalancerMaxNumberOfBundlesInBundleLoadReport=10
loadBalancerSplitIntervalMinutes=1
loadBalancerMaxNumberOfBundlesToSplitPerCycle=10
loadBalancerNamespaceBundleSplitConditionHitCountThreshold=3
loadBalancerServiceUnitStateTombstoneDelayTimeInSeconds=3600
loadBalancerMemoryResourceWeight=0
```
### What did you expect to see?
CPU usage converges to the same level with an amplitude not exceeding 5%
### What did you see instead?
There is a serious stratification problem in CPU usage, with the difference between the maximum usage and the minimum usage exceeding 40%. It looks like the load balancing feature isn't working correctly. I'm not sure if this is a bug or if I'm using it the wrong way.Please help me, thanks!
### Anything else?
If necessary, I can provide more information.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22911:
URL: https://github.com/apache/pulsar/issues/22911#issuecomment-2172407518
@AuroraTwinkle In the future, please create such questions in GitHub Discussions at https://github.com/apache/pulsar/discussions/categories/q-a . It's possible to migrate a question to an issue if that's needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22911:
URL: https://github.com/apache/pulsar/issues/22911#issuecomment-2172581929
> I'm not sure if this is a bug, so just close it first.
Most likely this could be considered a "feature" and not a bug. There are a lot of implementation level details that are undocumented. For example in this case, looking at the source code of ThresholdShedder could help:
https://github.com/apache/pulsar/blob/9692b67fd64f7acf90a93bf9adc21fab61842555/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ThresholdShedder.java#L77-L120
Enabling debug logging will be useful in following the decisions that were made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "AuroraTwinkle (via GitHub)" <gi...@apache.org>.
AuroraTwinkle closed issue #22911: [Bug] Pulsar broker CPU stratification problem
URL: https://github.com/apache/pulsar/issues/22911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "AuroraTwinkle (via GitHub)" <gi...@apache.org>.
AuroraTwinkle commented on issue #22911:
URL: https://github.com/apache/pulsar/issues/22911#issuecomment-2172513767
I'm not sure if this is a bug, so just close it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "AuroraTwinkle (via GitHub)" <gi...@apache.org>.
AuroraTwinkle commented on issue #22911:
URL: https://github.com/apache/pulsar/issues/22911#issuecomment-2172508390
> @AuroraTwinkle In the future, please create such questions in GitHub Discussions at https://github.com/apache/pulsar/discussions/categories/q-a . It's possible to migrate a question to an issue if that's needed.
Okay, got it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Pulsar broker CPU stratification problem [pulsar]
Posted by "AuroraTwinkle (via GitHub)" <gi...@apache.org>.
AuroraTwinkle commented on issue #22911:
URL: https://github.com/apache/pulsar/issues/22911#issuecomment-2173059457
> > I'm not sure if this is a bug, so just close it first.
>
> Most likely this could be considered a "feature" and not a bug. There are a lot of implementation level details that are undocumented. For example in this case, looking at the source code of ThresholdShedder could help:
>
> https://github.com/apache/pulsar/blob/9692b67fd64f7acf90a93bf9adc21fab61842555/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ThresholdShedder.java#L77-L120
>
>
> Enabling debug logging will be useful in following the decisions that were made.
Okay, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org