You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/05/18 13:32:41 UTC
[GitHub] [pulsar] frankjkelly opened a new issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
frankjkelly opened a new issue #10627:
URL: https://github.com/apache/pulsar/issues/10627
**Is your enhancement request related to a problem? Please describe.**
Add documentation for how to choose between load balancing algorithms
**Describe the solution you'd like**
* `supportedNamespaceBundleSplitAlgorithms` supports two algorithms for namespace bundle split `range_equally_divide,topic_count_equally_divide`
* Suggest adding some documentation around that to here https://pulsar.apache.org/docs/en/administration-load-balance/ including (1) documentation how each algorithm works and how each algorithm can be adjusted (any parameters to control operation) (2) documentation how and when to choose between the default `range_equally_divide` and `topic_count_equally_divide`
CC: @Jennifer88huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet edited a comment on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845588898
Currently, we have documented the usage in `broker.conf` as below, but do not provide more explanations in the user doc.
```
# Supported algorithms name for namespace bundle split.
# "range_equally_divide" divides the bundle into two parts with the same hash range size.
# "topic_count_equally_divide" divides the bundle into two parts with the same topics count.
supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #10627:
URL: https://github.com/apache/pulsar/issues/10627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
frankjkelly edited a comment on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845835933
Thanks @Anonymitaet exactly - it gets pretty interesting to understand how all those parameters will interact.
I'm particularly interested in how each algorithm behaves when we add brokers (we're hoping to implement Kubernetes autoscaling)- how quickly would those brokers be discovered and how quickly some load would be redirected to those new brokers.
In my limited experience with `range_equally_divide` it's not until we hit the `loadBalancerBrokerOverloadedThresholdPercentage` and the broker is overloaded that some namespace bundles will be split and offloaded but at that point usually the load on the broker is pretty high so that the offload seems to be "expensive" or at least take a long time and then our Pulsar Clients start timing out (we have them set with a 3 second timeout due to the near-real-time nature of our use-cases).
For us we'd rather have a way that was a faster balancing (but perhaps less precisely load balanced) than one that was more accurately load balanced but takes longer to shed load. Again we're pretty new to this but would appreciate to hear perspectives or pointers to code. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 closed issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 closed issue #10627:
URL: https://github.com/apache/pulsar/issues/10627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901007011
> I have a few questions:
>
> * How does each algorithm work with other [split bundle configurations](https://github.com/apache/pulsar/blob/b1ff552333c20b081e914766ba54a17037e3231f/conf/broker.conf#L988)?
> 
> * Is each algorithm limited by other split bundle configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901009444
@Anonymitaet each algorithm is affected by these configurations the same. `range_equally_divide` use hash range, `topic_count_equally_divide` is evenly distributed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845835933
Thanks @Anonymitaet exactly - it gets pretty interesting to understand how all those parameters will interact.
I'm particularly interested in how each algorithm behaves when we add brokers (we're hoping to implement Kubernetes autoscaling)- how quickly would those brokers be discovered and how quickly some load would be redirected to those new brokers.
In my limited experience with `range_equally_divide` it's not until we hit the `loadBalancerBrokerOverloadedThresholdPercentage` and the broker is overloaded that some namespace bundles will be split and offloaded but at that point usually the load on the broker is pretty high so that the offload seems to be "expensive" or at least take a long time and then our Pulsar Clients start timing out.
For us we'd rather have a way that was a faster balancing (but perhaps less precisely load balanced) than one that was more accurately load balanced but takes longer to shed load. Again we're pretty new to this but would appreciate to hear perspectives or pointers to code. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901007011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 removed a comment on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 removed a comment on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901007011
> I have a few questions:
>
> * How does each algorithm work with other [split bundle configurations](https://github.com/apache/pulsar/blob/b1ff552333c20b081e914766ba54a17037e3231f/conf/broker.conf#L988)?
> 
> * Is each algorithm limited by other split bundle configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-843701564
@frankjkelly thanks for raising this up. Would you like to add docs accordingly? Then we can help review, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 removed a comment on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 removed a comment on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901007011
> I have a few questions:
>
> * How does each algorithm work with other [split bundle configurations](https://github.com/apache/pulsar/blob/b1ff552333c20b081e914766ba54a17037e3231f/conf/broker.conf#L988)?
> 
> * Is each algorithm limited by other split bundle configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-844024703
@Anonymitaet I would love to but I really do not have the expertise to understand how these mechanisms work at a detailed level and to provide some guidance on how to choose between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845588898
@frankjkelly we have documented the usage in `broker.conf` as below:
```
# Supported algorithms name for namespace bundle split.
# "range_equally_divide" divides the bundle into two parts with the same hash range size.
# "topic_count_equally_divide" divides the bundle into two parts with the same topics count.
supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide
```
Does this work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 closed issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
congbobo184 closed issue #10627:
URL: https://github.com/apache/pulsar/issues/10627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845604444
I have a few questions:
- How does each algorithm work with other [split bundle configurations](https://github.com/apache/pulsar/blob/b1ff552333c20b081e914766ba54a17037e3231f/conf/broker.conf#L988)?
- Is each algorithm limited by other split bundle configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-901813304
Thanks @codelipenghui this is super helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet edited a comment on issue #10627: Documentation: Suggest adding documentation for supported bundle split algorithms
Posted by GitBox <gi...@apache.org>.
Anonymitaet edited a comment on issue #10627:
URL: https://github.com/apache/pulsar/issues/10627#issuecomment-845604444
I have a few questions:
- How does each algorithm work with other [split bundle configurations](https://github.com/apache/pulsar/blob/b1ff552333c20b081e914766ba54a17037e3231f/conf/broker.conf#L988)?
- Is each algorithm limited by other split bundle configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org