You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/14 07:11:50 UTC
[GitHub] [pulsar] andrasbeni opened a new pull request, #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
andrasbeni opened a new pull request, #16901:
URL: https://github.com/apache/pulsar/pull/16901
Fixes #15254
### Motivation
The number of topics in a namespace is limited by ZooKeeper when it is used as the local store. This is because listing topics requires all topic names to fit in ZK's buffer. By introducing an intermediate layer between the namespace node and topics, a namespace can have more topics.
### Modifications
- Add new namespace policy `number_of_buckets` defaulting to 1 (i.e. no buckets)
- Calculate topics' persistence name based on the number of buckets in the namespace
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
*(example:)*
- Added integration tests for creating topics in a bucketed namespace and producing to / consuming from it
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [X] `doc-required`
(Your PR needs to update docs and you will update later)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1235192935
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1289885886
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
URL: https://github.com/apache/pulsar/pull/16901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by "andrasbeni (via GitHub)" <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1485002375
@merlimat, I don't think I will have time to work on this anytime soon. Feel free to take over if this is still relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by "andrasbeni (via GitHub)" <gi...@apache.org>.
andrasbeni closed pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
URL: https://github.com/apache/pulsar/pull/16901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1249066788
@merlimat , please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on a diff in pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on code in PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#discussion_r938768857
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java:
##########
@@ -1219,7 +1220,11 @@ private CompletableFuture<List<String>> getPartitionsForTopic(TopicName topicNam
}
public CompletableFuture<List<String>> getListOfPersistentTopics(NamespaceName namespaceName) {
- return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName);
+ return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName)
+ .thenApply(x -> {
+ LOG.info("getListOfPersistentTopics: " + x);
Review Comment:
I don't think it needs to be logged at all. Reverted this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1244117496
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1245079540
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nahguam commented on a diff in pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
nahguam commented on code in PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#discussion_r937848649
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java:
##########
@@ -1219,7 +1220,11 @@ private CompletableFuture<List<String>> getPartitionsForTopic(TopicName topicNam
}
public CompletableFuture<List<String>> getListOfPersistentTopics(NamespaceName namespaceName) {
- return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName);
+ return pulsar.getPulsarResources().getTopicResources().listPersistentTopicsAsync(namespaceName)
+ .thenApply(x -> {
+ LOG.info("getListOfPersistentTopics: " + x);
Review Comment:
Does this need to be info?
(also `x` -> `topics`)
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/TopicResources.java:
##########
@@ -41,22 +42,29 @@ public class TopicResources {
private static final String MANAGED_LEDGER_PATH = "/managed-ledgers";
private final MetadataStore store;
+ private final NamespaceResources namespaceResources;
private final Map<BiConsumer<String, NotificationType>, Pattern> topicListeners;
- public TopicResources(MetadataStore store) {
+ public TopicResources(MetadataStore store, NamespaceResources namespaceResources) {
this.store = store;
+ this.namespaceResources = namespaceResources;
topicListeners = new ConcurrentHashMap<>();
store.registerListener(this::handleNotification);
}
public CompletableFuture<List<String>> listPersistentTopicsAsync(NamespaceName ns) {
String path = MANAGED_LEDGER_PATH + "/" + ns + "/persistent";
- return store.getChildren(path).thenApply(children ->
- children.stream().map(c -> TopicName.get(TopicDomain.persistent.toString(), ns, decode(c)).toString())
- .collect(Collectors.toList())
- );
+ CompletableFuture<List<String>> childrenOfNamespace = store.getChildren(path);
+ CompletableFuture<List<String>> topics = childrenOfNamespace.thenCompose(children ->
+ !children.isEmpty() && children.get(0).startsWith(TopicName.BUCKET_ID_PERFIX)
+ ? children.stream().map(bucket -> store.getChildren(path + "/" + bucket))
+ .collect(FutureUtil.toList())
+ : CompletableFuture.completedFuture(children));
Review Comment:
This looks identical to L76-81, could they share a common helper method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on a diff in pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on code in PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#discussion_r938769579
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/TopicResources.java:
##########
@@ -41,22 +42,29 @@ public class TopicResources {
private static final String MANAGED_LEDGER_PATH = "/managed-ledgers";
private final MetadataStore store;
+ private final NamespaceResources namespaceResources;
private final Map<BiConsumer<String, NotificationType>, Pattern> topicListeners;
- public TopicResources(MetadataStore store) {
+ public TopicResources(MetadataStore store, NamespaceResources namespaceResources) {
this.store = store;
+ this.namespaceResources = namespaceResources;
topicListeners = new ConcurrentHashMap<>();
store.registerListener(this::handleNotification);
}
public CompletableFuture<List<String>> listPersistentTopicsAsync(NamespaceName ns) {
String path = MANAGED_LEDGER_PATH + "/" + ns + "/persistent";
- return store.getChildren(path).thenApply(children ->
- children.stream().map(c -> TopicName.get(TopicDomain.persistent.toString(), ns, decode(c)).toString())
- .collect(Collectors.toList())
- );
+ CompletableFuture<List<String>> childrenOfNamespace = store.getChildren(path);
+ CompletableFuture<List<String>> topics = childrenOfNamespace.thenCompose(children ->
+ !children.isEmpty() && children.get(0).startsWith(TopicName.BUCKET_ID_PERFIX)
+ ? children.stream().map(bucket -> store.getChildren(path + "/" + bucket))
+ .collect(FutureUtil.toList())
+ : CompletableFuture.completedFuture(children));
Review Comment:
Nice catch. I did not realize this. I extracted the common part into a method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1242740832
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1244115674
/pulsarbot rerun pulsar-ci-checks-completed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1243959387
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1529773618
Thanks for your contribution @andrasbeni!
It seems the community doesn't give this PR a review in time. I add this PR in my backlog to see if it's still relevant. Perhaps ping you for reviewing if so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1202171218
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1256989462
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] andrasbeni commented on pull request #16901: [feat][broker] PIP-157 Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on PR #16901:
URL: https://github.com/apache/pulsar/pull/16901#issuecomment-1225891594
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org