You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/09 15:23:39 UTC
[GitHub] [pulsar] tongsucn opened a new issue #8492: Unloaded namespace bundles may not be assigned to suitable brokers.
tongsucn opened a new issue #8492:
URL: https://github.com/apache/pulsar/issues/8492
After unloading namespace bundle, one of the available brokers shall be selected for the unloaded bundle to continue working. All available broker candidates will be filtered in `org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl.selectBrokerForAssignment`, it:
1. applies NS policy if any, then,
2. filters out brokers with too many topics from candidates, then,
3. filters out brokers based on anit-affinity-group setting, then,
4. **removes brokers that owning non-minimum-number of bundles, then,**
5. filters out brokers that should be ignored (e.g. old version broker), then,
6. selects the best broker that having lowest load.
The problem happens at step 4, it only keeps the broker candidates that having least bundles during the filtering. That is, the broker selection may be completely based on bunlde number on the broker, rather than the actual load.
For example, on this cluster (`2.6.1` with 8 brokers), 8 topics (partitions) have significantly higher OUT throughput than the others. Ideally they should be evenly assigned on the brokers, rather than working on a small subset of the brokers. However, the filtering rule based on bundle number converges these 8 topics on 2 brokers (4 on each), which results in these 2 brokers having much higher OUT throughput, but much lower amount of bunles than the rest.
Although the `ThresholdShedding` from #6772 is able to detect that these 2 brokers are overloaded, they cannot be moved to the rest brokers because that only these 2 brokers can "survive" after step 4. The consequence is that these 8 topics are repeatly moved between these two brokers during load-balancing.
I think there should be at least two solutions:
1. Just remove the step 4, because the other steps, especailly the last one, should be enough to find out the suitable broker, i.e. the broker with lowest load.
2. Make this configurable, such as adding a new config called `loadBalancePreferBrokersWithLeastBundles`.
How do you like it? Or are there any other, more graceful, solutions to this problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tongsucn commented on issue #8492: Unloaded namespace bundles may not be assigned to suitable brokers.
Posted by GitBox <gi...@apache.org>.
tongsucn commented on issue #8492:
URL: https://github.com/apache/pulsar/issues/8492#issuecomment-726089500
> I think you can try to use`topic_count_equally_divide` for the defaultNamespaceBundleSplitAlgorithm, the default policy is `range_equally_divide`, when there are few topics, the `topic_count_equally_divide` is better than `range_equally_divide`
@codelipenghui Thank for your reply! I just changed the configuration you suggested, let me observe it for a while, thanks~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tongsucn commented on issue #8492: Unloaded namespace bundles may not be assigned to suitable brokers.
Posted by GitBox <gi...@apache.org>.
tongsucn commented on issue #8492:
URL: https://github.com/apache/pulsar/issues/8492#issuecomment-734158045
@codelipenghui I tried the config you gave me, but the issue persists. But I tried split the big topic into more partitions, the situation is better now. Thanks anyway~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #8492: Unloaded namespace bundles may not be assigned to suitable brokers.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #8492:
URL: https://github.com/apache/pulsar/issues/8492#issuecomment-725383862
I think you can try to use`topic_count_equally_divide` for the defaultNamespaceBundleSplitAlgorithm, the default policy is `range_equally_divide`, when there are few topics, the `topic_count_equally_divide` is better than `range_equally_divide`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tongsucn closed issue #8492: Unloaded namespace bundles may not be assigned to suitable brokers.
Posted by GitBox <gi...@apache.org>.
tongsucn closed issue #8492:
URL: https://github.com/apache/pulsar/issues/8492
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org