You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/21 13:13:51 UTC
[GitHub] [pulsar] andrasbeni opened a new issue, #15254: PIP-157: Bucketing topic metadata to allow more topics per namespace
andrasbeni opened a new issue, #15254:
URL: https://github.com/apache/pulsar/issues/15254
<!---
Instructions for creating a PIP using this issue template:
1. The author(s) of the proposal will create a GitHub issue ticket using this template.
(Optionally, it can be helpful to send a note discussing the proposal to
dev@pulsar.apache.org mailing list before submitting this GitHub issue. This discussion can
help developers gauge interest in the proposed changes before formalizing the proposal.)
2. The author(s) will send a note to the dev@pulsar.apache.org mailing list
to start the discussion, using subject prefix `[PIP] xxx`. To determine the appropriate PIP
number `xxx`, inspect the mailing list (https://lists.apache.org/list.html?dev@pulsar.apache.org)
for the most recent PIP. Add 1 to that PIP's number to get your PIP's number.
3. Based on the discussion and feedback, some changes might be applied by
the author(s) to the text of the proposal.
4. Once some consensus is reached, there will be a vote to formally approve
the proposal. The vote will be held on the dev@pulsar.apache.org mailing list. Everyone
is welcome to vote on the proposal, though it will considered to be binding
only the vote of PMC members. It will be required to have a lazy majority of
at least 3 binding +1s votes. The vote should stay open for at least 48 hours.
5. When the vote is closed, if the outcome is positive, the state of the
proposal is updated and the Pull Requests associated with this proposal can
start to get merged into the master branch.
-->
## Motivation
Pulsar is able to manage millions of topics but the number of topics within a single namespace is limited by metadata storage.
For each topic within a namespace there is a ZooKeeper node. Listing topics thus requires listing children of a node, which at around 10K topics hits the limits of ZK.
## Goal
This feature will allow larger number of topics within a namespace by inserting an intermediate layer (buckets) before the topic nodes like `/managed-ledgers/tenant/namespace/domain/bucket/topic`.
By default this feature will be switched off and would only be enabled on a per namespace basis at the creation of namespaces by setting a policy. This eliminates the need for migrating existing installations to this new scheme.
Buckets will not have correlation with bundles.
## API Changes
A new policy `numberOfTopicBuckets` will be added. The default value, 1 means no bucketing, the current behaviour will be preserved for the namespace. Greater values mean topics will be stored at a path including buckets. Users will not be able to change the number of buckets after the namespace is created.
## Implementation
The goal is to implement this feature transparently to the user. Clients will continue to refer to topics by domain://tenant/namespace/topic but pulsar will internally translate to the new persistence naming where necessary.
The way metadata stores work will not be affected either.
Assigning topics to buckets will be based on the topic name's hash code's absolute value modulo the number of buckets.
The bulk of the changes necessary for this feature is to make namespace policies available wherever persistence naming is calculated. Where listing of topics within a namespace is necessary, the introduction of the new layer will add some overhead in the form of multiple requests to the metadata store. These include checking checking if the limit on topic number per namespace has been reached.
### Example
Let's consider the following metadata hierarchy:
```
managed-ledgers
\- tenant
\- namespace
\- persistent
+- nptopic1
+- nptopic2
+- ptopic-partition-0
+- ptopic-partition-1
+- ptopic-partition-2
\- ptopic-partition-3
```
In case of 3 buckets the same topic metadata would be laid out the following way:
```
managed-ledgers
\- tenant
\- namespace
\- persistent
+- $0
| +- ptopic-partition-0
| \- ptopic-partition-3
+- $1
| +- nptopic2
| \- ptopic-partition-1
\- $2
+- nptopic1
\- ptopic-partition-2
```
### Compatibility
Existing namespaces and namespaces created without explicitly activating this feature will not be affected.
Namespaces created with this feature activated can be used just as others.
## Rejected alternatives
An alternative approach would be to introduce bucketing globally for all namespaces. This would make metadata structure more homogeneous but would require complex update logic to atomically move topics from their current path to the new place once all brokers are upgraded.
For similar reasons changing the number of buckets is not a goal of this proposal.
Since the proposal intends to solve a problem related to ZK, it could be handled within the ZK based metadata store implementation. This would have to introduce knowledge of what paths mean thus breaking separation of concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #15254: PIP-157: Bucketing topic metadata to allow more topics per namespace
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15254:
URL: https://github.com/apache/pulsar/issues/15254#issuecomment-1145518438
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org