You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/23 09:44:15 UTC
[GitHub] [pulsar] gaozhangmin opened a new pull request #13470: pluggable-consumer-selector
gaozhangmin opened a new pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470
Fix #13347
Support user defined and pluggable consumer selector by adding a broker configuration `keySharedConsumerSelectorStrategy` which default value is `org.apache.pulsar.broker.service.ConsistentHashingStickyKeyConsumerSelector`
### Modification
Currently, client can specify keySharedPolicy like this, but only support pre-defined `autoSplitHashRange` and `stickyHashRange`.
https://github.com/apache/pulsar/blob/6c7dcc0cf877cfcb8bcea18cde7662ebacb01d4c/pulsar-broker/src/test/java/org/apache/pulsar/client/api/KeySharedSubscriptionTest.java#L1034-L1041
Now, custom keySharedPolicy `KeySharedPolicyCustome` is supported, and the pre configured `keySharedConsumerSelectorStrategy` will be used.
If client doesn't have a keySharedPolicy, `KeySharedMode.CUSTOME` will be used.
### Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [ ] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13470: [feature] Pluggable consumer selector for key shared subscription type
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1048730833
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #13470: [feature] Pluggable consumer selector for key shared subscription type
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1033462208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin closed pull request #13470: [feature] Pluggable consumer selector for key shared subscription type
Posted by GitBox <gi...@apache.org>.
gaozhangmin closed pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1033462208
> Maintaining such extensions will be very hard for users as we are deep inside the internals of Pulsar.
>
> We should define a smaller surface for the extension point, like extracting the only part that selects the consumers
@Jason918 @codelipenghui @315157973 For this review, Do we have better design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1000173913
@gaozhangmin:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1000175966
@gaozhangmin:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#discussion_r785399892
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractDispatcherMultipleConsumers.java
##########
@@ -239,5 +239,24 @@ private int getFirstConsumerIndexOfPriority(int targetPriority) {
private static final Logger log = LoggerFactory.getLogger(PersistentStickyKeyDispatcherMultipleConsumers.class);
+ public StickyKeyConsumerSelector createConsumerSelectorInstance(ServiceConfiguration conf) {
+ try {
+ Class<?> consumerSelectorClass = Class.forName(conf.getKeySharedConsumerSelectorStrategy());
+ Object consumerSelectorClassInstance = consumerSelectorClass.getDeclaredConstructor().newInstance();
+ if (consumerSelectorClassInstance instanceof StickyKeyConsumerSelector) {
+ return (StickyKeyConsumerSelector) consumerSelectorClassInstance;
+ } else {
+ log.error("create key shared consumer selector strategy failed."
Review comment:
We cannot fallback to another algorithm otherwise the system will break as all the assumptions in the application won't be guaranteed.
Let's throw a bad error here and in other failure cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin removed a comment on pull request #13470: [feature] Pluggable consumer selector for key shared subscription type
Posted by GitBox <gi...@apache.org>.
gaozhangmin removed a comment on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1048635763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13470: [feature] Pluggable consumer selector for key shared subscription type
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1048635763
> Maintaining such extensions will be very hard for users as we are deep inside the internals of Pulsar.
>
> We should define a smaller surface for the extension point, like extracting the only part that selects the consumers
Do you have better design? @eolivelli .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaozhangmin commented on pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1018277392
> Maintaining such extensions will be very hard for users as we are deep inside the internals of Pulsar.
+1, I will work it around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #13470: pluggable-consumer-selector
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #13470:
URL: https://github.com/apache/pulsar/pull/13470#issuecomment-1000265344
This is a good feature, but it needs to be discussed and voted. Please send an email first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org