You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/16 20:25:11 UTC
[PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
merlimat opened a new pull request, #26953:
URL: https://github.com/apache/pulsar/pull/26953
Stacked on #26949: its two commits are included here until it merges. Only the last commit, "[fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic", is new.
### Problem
Migrating a regular topic to a scalable topic (PIP-475) leaves its topic-level permissions on `persistent://t/n/x`. With #26949, the scalable topic and its `segment://` segments are authorized with the permissions of `topic://t/n/x`, and the migration grants nothing there. A role allowed only on the regular topic loses its access when the migration commits:
- the V5 producers it was running fail on their next send, as they move to the new segments;
- a V5 client that uses `topic://t/n/x` can't create a producer or a consumer.
Roles granted on the namespace are not affected.
### Example
```
pulsar-admin topics create-partitioned-topic persistent://public/default/orders -p 2
pulsar-admin topics grant-permission persistent://public/default/orders --role app --actions produce,consume
# "app" runs a V5 producer on persistent://public/default/orders
pulsar-admin scalable-topics migrate persistent://public/default/orders
```
The producer's next send fails:
```
org.apache.pulsar.client.api.v5.PulsarClientException: {"errorMsg":"Client is not authorized to Lookup", ...}
at org.apache.pulsar.client.impl.v5.ScalableTopicProducer.getOrCreateSegmentProducer(ScalableTopicProducer.java:1000)
```
`ScalableTopics.doMigrateToScalableAsync` creates the new segments, commits the scalable metadata and terminates the old topics, without touching permissions:
```java
return createMigratedChildTopicsAsync(scalableName, metadata)
.thenCompose(__ -> resources().createScalableTopicAsync(scalableName, metadata))
.thenCompose(__ -> terminateLegacyTopicsAsync(persistentBase, partitions));
```
The producer then moves to a new segment, `segment://public/default/orders/...`, which `PulsarAuthorizationProvider.checkPermission` authorizes with the permissions of `topic://public/default/orders`: there are none.
### Change
**The migration grants the regular topic's permissions on the scalable topic.** It does so after creating the new segments and before the metadata commit, so the V5 clients that follow the layout change are authorized on the new segments as soon as they see them:
```java
return createMigratedChildTopicsAsync(scalableName, metadata)
.thenCompose(__ -> getAuthorizationService().copyPermissionsAsync(persistentBase, scalableName))
.thenCompose(__ -> resources().createScalableTopicAsync(scalableName, metadata))
.thenCompose(__ -> terminateLegacyTopicsAsync(persistentBase, partitions));
```
- Only the permissions granted on `persistent://t/n/x` itself are copied:
- not the namespace permissions: they already apply to both topics, and as permissions of the scalable topic they would outlive their revocation on the namespace;
- not those granted on a single partition (`persistent://t/n/x-partition-K`): every new segment receives keys from all the partitions, so they would widen to the whole topic. The legacy segment that wraps the partition keeps honoring them.
- A role keeps the permissions already granted to it on `topic://t/n/x` (possible with `allowAclChangesOnNonExistentTopics`), and gains those granted to it on `persistent://t/n/x`.
- After the migration, each name keeps its own permissions: nothing syncs them, and `topic://` doesn't fall back to `persistent://`, as the two are distinct identities (scalable-topics spec, Compatibility §2). The permissions of `persistent://t/n/x` keep authorizing the legacy segments, which hold the data published before the migration.
- A failure before the commit leaves the copied permissions on `topic://t/n/x`; a retried migration adds them again.
**New `AuthorizationProvider.copyPermissionsAsync(sourceTopicName, targetTopicName)`.** Only the provider knows which permissions are granted on a topic itself rather than on its namespace (`getPermissionsAsync` returns both), so the copy is a provider method, exposed by `AuthorizationService`:
- The default implementation copies nothing, so third-party providers keep their current behavior.
- `PulsarAuthorizationProvider` implements it with one update of the namespace policies, adding each role's actions on the source topic to those it has on the target. It writes nothing when the source topic has no permissions of its own, so migrating such a topic doesn't require writable policies. With permissions to copy and read-only policies, the migration fails rather than lose them.
- `MultiRolesTokenAuthorizationProvider` inherits it.
**PIP-475** gains the new migration step (in both step lists), a Security Considerations section on what happens to the permissions, and the new provider method under its public-facing changes.
**Compatibility.** A new default method on the `AuthorizationProvider` plugin interface. No changes to the wire protocol, the REST API or metadata formats.
**Not in this PR:** after the migration, a V5 client that keeps addressing the topic as `persistent://t/n/x` is still authorized for its lookup and subscription with the permissions of `persistent://t/n/x`, the name it sends. Taking a role's access away entirely therefore takes a revocation on both names, as PIP-475 now documents.
### Testing
- New `ScalableTopicMigrationPermissionsTest` (token authentication, `PulsarAuthorizationProvider`, V5 client), with a role granted only on the regular topic:
- a producer created before the migration keeps sending after it, and a consumer then receives the messages of both the legacy segments and the new ones;
- after migrating a non-partitioned topic, the role produces and consumes through `topic://`;
- the role's permissions are copied, but not the namespace permissions of another role, nor the permissions granted on a single partition.
- Without the copy step, all three fail: the send after the migration and the `topic://` producer get `Client is not authorized to Lookup`, and `topic://` has no permissions.
- `PulsarAuthorizationProviderTest`, 3 new tests: merging with the permissions already on the target; nothing written when there is nothing to copy, even with read-only policies; failure with permissions to copy and read-only policies.
- These suites pass: `ScalableTopicPermissionsTest`, `ScalableTopicsAuthZTest`, `ScalableTopicMigrationTest`, `ScalableTopicsMigrateTest`, `V5MigrationEndToEndTest`, `ScalableTopicLegacySegmentTest`.
- `./gradlew quickCheck` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4300625476
##########
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/ScalableTopicsImpl.java:
##########
@@ -190,6 +192,53 @@ public CompletableFuture<Void> removeAutoScalePolicyAsync(String topic) {
return asyncDeleteRequest(topicPath(tn).path("autoScalePolicy"));
}
+ // --- Permissions ---
+
+ @Override
+ public Map<String, Set<AuthAction>> getPermissions(String topic) throws PulsarAdminException {
+ return sync(() -> getPermissionsAsync(topic));
+ }
+
+ @Override
+ public CompletableFuture<Map<String, Set<AuthAction>>> getPermissionsAsync(String topic) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncGetRequest(topicPath(tn).path("permissions"),
+ new GenericType<Map<String, Set<AuthAction>>>() {});
+ });
+ }
+
+ @Override
+ public void grantPermission(String topic, String role, Set<AuthAction> actions) throws PulsarAdminException {
+ sync(() -> grantPermissionAsync(topic, role, actions));
+ }
+
+ @Override
+ public CompletableFuture<Void> grantPermissionAsync(String topic, String role, Set<AuthAction> actions) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncPostRequest(permissionsPath(tn, role), Entity.entity(actions, MediaType.APPLICATION_JSON));
+ });
+ }
+
+ @Override
+ public void revokePermissions(String topic, String role) throws PulsarAdminException {
+ sync(() -> revokePermissionsAsync(topic, role));
+ }
+
+ @Override
+ public CompletableFuture<Void> revokePermissionsAsync(String topic, String role) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncDeleteRequest(permissionsPath(tn, role));
+ });
+ }
+
+ /** The role is a single path component, even if it contains a '/', as in the regular-topic permission API. */
+ private WebTarget permissionsPath(TopicName tn, String role) {
+ return WebTargets.addParts(topicPath(tn).path("permissions"), new String[] {role});
Review Comment:
Agreed, but not changed here: this is the admin client code from #26949, merged since, so it's no longer in this PR's diff after merging master. It also predates #26949: `ScalableTopicsImpl.topicPath` appends the raw local name for every scalable-topic call, not only the permission ones, so it needs its own fix, with reserved-character coverage across those calls and their CLI commands.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4333355538
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +299,51 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>The lock is an ephemeral record, so a broker that goes away while holding it doesn't keep it.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn, Supplier<CompletableFuture<T>> action) {
+ String path = joinPath(CREATION_LOCK_PATH, tn.getNamespace(), tn.getEncodedLocalName());
+ return acquireCreationLock(path, 1).thenCompose(lock -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(action);
+ return result.handle((__, ___) -> null)
+ .thenCompose(__ -> getStore().delete(path, Optional.of(lock.getVersion())))
Review Comment:
Fixed in 8d628ba728: each acquisition now creates an ephemeral record of its own, `/topic-creation-locks/{tenant}/{namespace}/{topic}:{token}` with a random token, and holds the lock only if no other record of the topic exists (listed from the store, not the cache; of two acquisitions, the second one to create its record sees the other). Releasing deletes that record only, so A's release after its session expired can't touch B's. The migration also gets a `CreationLock` and calls `checkHeld()` (its record is still there, owned by the current session) right before granting the permissions and right before the metadata commit, so a continuation that lost the lock stops there. `ScalableTopicCreationLockTest.testLockLostWithTheMetadataSession` replays your scenario on a real ZooKeeper (expire A's session, B takes the lock, A completes): A's `checkHeld()` fails, and B keeps the lock. Not covered: a write the store already sent and retries across the session expiry, which would need fen
cing in the store itself.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +299,51 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>The lock is an ephemeral record, so a broker that goes away while holding it doesn't keep it.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn, Supplier<CompletableFuture<T>> action) {
+ String path = joinPath(CREATION_LOCK_PATH, tn.getNamespace(), tn.getEncodedLocalName());
+ return acquireCreationLock(path, 1).thenCompose(lock -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(action);
+ return result.handle((__, ___) -> null)
+ .thenCompose(__ -> getStore().delete(path, Optional.of(lock.getVersion())))
+ .exceptionally(ex -> {
+ // It goes away with the broker's metadata session anyway.
+ log.warn().attr("path", path).exceptionMessage(ex).log("Failed to release a creation lock");
+ return null;
+ })
+ .thenCompose(__ -> result);
+ });
+ }
+
+ private CompletableFuture<Stat> acquireCreationLock(String path, int attempt) {
+ return getStore().put(path, new byte[0], Optional.of(-1L), Set.of(Option.Ephemeral.INSTANCE))
+ .exceptionallyCompose(ex -> {
+ if (!(FutureUtil.unwrapCompletionException(ex)
+ instanceof MetadataStoreException.BadVersionException)) {
+ return CompletableFuture.failedFuture(ex);
+ }
+ if (attempt == CREATION_LOCK_ATTEMPTS) {
+ return CompletableFuture.failedFuture(new MetadataStoreException.LockBusyException(
+ "Another creation or migration of the scalable topic is in progress: " + path));
Review Comment:
Fixed in 8d628ba728: each acquisition's record is named after its own random token, so a `BadVersion` on its create (the store's retry finding the record after a lost response) can only mean the record is this acquisition's own, and it holds the lock instead of waiting on itself; no other request's record has that name. Any other create failure deletes the record, in case it was created anyway. `ScalableTopicCreationLockTest.testAcquisitionWhoseCreateResponseWasLost` commits the record and then reports `BadVersion`: the acquisition holds the lock, releases it, and the next one takes it.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #26953:
URL: https://github.com/apache/pulsar/pull/26953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4333283662
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +299,51 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>The lock is an ephemeral record, so a broker that goes away while holding it doesn't keep it.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn, Supplier<CompletableFuture<T>> action) {
+ String path = joinPath(CREATION_LOCK_PATH, tn.getNamespace(), tn.getEncodedLocalName());
+ return acquireCreationLock(path, 1).thenCompose(lock -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(action);
+ return result.handle((__, ___) -> null)
+ .thenCompose(__ -> getStore().delete(path, Optional.of(lock.getVersion())))
Review Comment:
**[P1] Do not release a successor's lock after session expiration**
If A's metadata session expires while its action is pending, B can acquire this path before A reconnects and completes. ZooKeeper initializes both node incarnations at data version 0, so this delete removes B's active lock. The default broker policy is `reconnect`, and the pending migration continuation is not cancelled or fenced by the session-event handlers.
I reproduced the deletion on this head using a real ZooKeeper server and two metadata clients: expire A's session, wait for reconnection, acquire and hold the replacement with B, then complete A. The assertion that B's lock remains present fails because the node is gone. This directly demonstrates lost exclusion; the resulting permission-copy exposure is source-backed rather than a full two-broker migration reproduction.
Please make release acquisition/session-aware and prevent stale migration continuations from performing protected side effects after ownership loss. A resettable data version is not an acquisition identity, and checking a token only during cleanup would not fence the pending action. Add session-expiry/replacement regression coverage.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +299,51 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>The lock is an ephemeral record, so a broker that goes away while holding it doesn't keep it.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn, Supplier<CompletableFuture<T>> action) {
+ String path = joinPath(CREATION_LOCK_PATH, tn.getNamespace(), tn.getEncodedLocalName());
+ return acquireCreationLock(path, 1).thenCompose(lock -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(action);
+ return result.handle((__, ___) -> null)
+ .thenCompose(__ -> getStore().delete(path, Optional.of(lock.getVersion())))
+ .exceptionally(ex -> {
+ // It goes away with the broker's metadata session anyway.
+ log.warn().attr("path", path).exceptionMessage(ex).log("Failed to release a creation lock");
+ return null;
+ })
+ .thenCompose(__ -> result);
+ });
+ }
+
+ private CompletableFuture<Stat> acquireCreationLock(String path, int attempt) {
+ return getStore().put(path, new byte[0], Optional.of(-1L), Set.of(Option.Ephemeral.INSTANCE))
+ .exceptionallyCompose(ex -> {
+ if (!(FutureUtil.unwrapCompletionException(ex)
+ instanceof MetadataStoreException.BadVersionException)) {
+ return CompletableFuture.failedFuture(ex);
+ }
+ if (attempt == CREATION_LOCK_ATTEMPTS) {
+ return CompletableFuture.failedFuture(new MetadataStoreException.LockBusyException(
+ "Another creation or migration of the scalable topic is in progress: " + path));
Review Comment:
**[P2] Reconcile an acquisition whose create committed but acknowledgement was lost**
`ZKMetadataStore` explicitly retries multi operations after `CONNECTIONLOSS`. If this ephemeral create committed before the reply was lost, the retry sees `NODEEXISTS`, which the metadata layer maps to `BadVersionException`. This code treats that as another operation's lock, exhausts retries, and never enters the cleanup path—even though its own session still owns the node. Later create/migrate attempts for this topic can therefore keep returning `LockBusy` until session termination or cleanup.
A local regression on this head committed the real ZooKeeper node, injected the `BadVersion` outcome of that lost-response retry, and then attempted a separate acquisition. Both attempts returned `LockBusy`. This is fault injection of a reachable metadata outcome, not an actual packet-loss reproduction.
Please retain a unique per-acquisition token across retries and reconcile ambiguous outcomes, recovering or safely cleaning up that specific acquisition. Do not adopt another request's lock solely because it belongs to the same metadata session. Add a lost-acknowledgement recovery test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4343230607
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/ScalableTopics.java:
##########
@@ -354,9 +384,20 @@ private CompletableFuture<Void> doMigrateToScalableAsync(TopicName scalableName,
ScalableTopicController.createMigratedMetadata(persistentBase, partitions,
pulsar().getConfiguration().getScalableTopicEntryBucketBudget(),
pulsar().getConfiguration().getScalableTopicEntryBucketMaxPerSegment());
+ // The new segments are authorized with the permissions of the scalable topic, a distinct
+ // identity, while the legacy segments keep those of the persistent:// topics they wrap. So
+ // grant the regular topic's permissions on the scalable topic before the commit: the V5
+ // clients that follow the layout change are then authorized on the new segments as soon as
+ // they see them. Namespace permissions apply to both topics already. Those of a single
+ // partition are not copied: the new segments mix its data with the other partitions' data.
+ // Both steps check that the creation lock is still held first: a broker that lost it, as when
+ // its metadata session expired during the steps above, must not make them.
return createMigratedChildTopicsAsync(scalableName, metadata)
+ .thenCompose(__ -> lock.checkHeld())
+ .thenCompose(__ -> getAuthorizationService().copyPermissionsAsync(persistentBase, scalableName))
+ .thenCompose(__ -> lock.checkHeld())
Review Comment:
**[P1] A successful preflight check does not fence the permission write in flight**
The first `checkHeld()` can succeed, then the real provider can wait on its configuration-store policy refresh/read-only operations before issuing the namespace update. If the **local** metadata session expires during that wait, another request can acquire the free creation lock and create the independent target. When the pending configuration operation completes, the provider still copies the source grants onto that target. The second check rejects the migration only after those grants persist.
This is not limited to a write already sent and retried by the local store: the configuration and local stores can use independent sessions, and the namespace mutation can start after local ownership was lost. The provider receives no acquisition identity with which to fence it.
Please protect the actual permission mutation/target identity across ownership loss, rather than relying solely on preflight reads—for example, a durable reservation/generation protocol preventing an independent creation while an earlier permission operation can still complete. Add coverage that expires the local session during the real provider's configuration operation, after the first check passes.
**Validation limit:** this sequence is source-backed. My attempted broker reproduction stopped at a post-expiry segment-ownership HTTP 412 before the ACL assertion; I am not claiming an observed end-to-end ACL failure on this revision.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +301,101 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /** The creation lock of a scalable topic, while {@link #runWithCreationLock} holds it. */
+ public interface CreationLock {
+ /**
+ * Complete normally if the lock is still held, or fail with {@link MetadataStoreException.LockBusyException}.
+ * An action checks it before each change the lock guards: a broker whose metadata session expired lost the
+ * lock, which another broker may have taken since.
+ */
+ CompletableFuture<Void> checkHeld();
+ }
+
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>Each acquisition creates an ephemeral record of its own, named after the topic and a random token, and
+ * holds the lock if no other record of the topic exists then. So a broker that goes away doesn't keep the lock;
+ * releasing it deletes that record only, never the one of an acquisition made after this broker's metadata
+ * session expired; and a create that the store retried after its response was lost finds its own record.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn,
+ Function<CreationLock, CompletableFuture<T>> action) {
+ String lockDir = joinPath(CREATION_LOCK_PATH, tn.getNamespace());
+ String prefix = tn.getEncodedLocalName() + ":";
+ String name = prefix + UUID.randomUUID();
+ String path = joinPath(lockDir, name);
+ CreationLock lock = () -> getStore().get(path).thenAccept(record -> {
+ if (record.isEmpty() || !record.get().getStat().isCreatedBySelf()) {
+ throw new CompletionException(new MetadataStoreException.LockBusyException(
+ "Lost the creation lock of " + tn));
+ }
+ });
+ return acquireCreationLock(lockDir, prefix, name, 1).thenCompose(__ -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(() -> action.apply(lock));
+ return result.handle((___, ____) -> null)
+ .thenCompose(___ -> releaseCreationLock(path))
+ .thenCompose(___ -> result);
+ });
+ }
+
+ private CompletableFuture<Void> acquireCreationLock(String lockDir, String prefix, String name, int attempt) {
+ String path = joinPath(lockDir, name);
+ return getStore().put(path, new byte[0], Optional.of(-1L), Set.of(Option.Ephemeral.INSTANCE))
+ .<Void>thenApply(__ -> null)
+ .exceptionallyCompose(ex -> {
+ if (FutureUtil.unwrapCompletionException(ex)
+ instanceof MetadataStoreException.BadVersionException) {
+ // The record exists: the store retried a create of this acquisition whose response was
+ // lost. No other acquisition uses this name.
+ return CompletableFuture.completedFuture(null);
+ }
+ // Delete the record, in case it was created anyway.
+ return releaseCreationLock(path).thenCompose(__ -> CompletableFuture.failedFuture(ex));
+ })
+ // Read the store, not the cache, which may miss the record of a concurrent acquisition.
+ .thenCompose(__ -> getStore().getChildrenFromStore(lockDir))
Review Comment:
**[P2] Clean up the acquisition record if listing contenders fails**
This stage runs after the UUID record has been created, but its failure is outside the preceding `exceptionallyCompose` cleanup. It also prevents acquisition from completing, so the action-completion cleanup in `runWithCreationLock` is never installed. One failed contender read therefore leaves an abandoned ephemeral record; subsequent requests see it as a competing acquisition and return `LockBusy` until session termination or manual cleanup.
I reproduced this on the reviewed head by allowing the real `LocalMemoryMetadataStore` put to complete, failing only the first `getChildrenFromStore` call, and retrying on the same live store. The first action never ran, its `orders:<token>` record remained, and the next independent acquisition failed with `LockBusy`.
Please release the acquisition's own record on every terminal acquisition failure after it may have been created, including a contender-list failure, while preserving the original error and leaving successful acquisitions owned until their actions finish. Add a failed-list-then-reacquire regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4300624717
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java:
##########
@@ -840,6 +847,53 @@ public CompletableFuture<Void> removePermissionsAsync(TopicName topicName) {
});
}
+ @Override
+ public CompletableFuture<Void> copyPermissionsAsync(TopicName sourceTopicName, TopicName targetTopicName) {
+ return pulsarResources.getNamespaceResources().getPoliciesAsync(sourceTopicName.getNamespaceObject())
+ .thenCompose(sourcePolicies -> {
+ Map<String, Set<AuthAction>> sourceRoles = sourcePolicies
+ .map(policies -> policies.auth_policies.getTopicAuthentication()
+ .get(sourceTopicName.toString()))
+ .orElse(null);
+ if (sourceRoles == null || sourceRoles.isEmpty()) {
+ // Nothing to copy: don't require the policies to be writable.
+ return CompletableFuture.completedFuture(null);
+ }
+ return getPoliciesReadOnlyAsync().thenCompose(readonly -> {
+ if (readonly) {
+ log.debug("Policies are read-only. Broker cannot do read-write operations");
+ throw new IllegalStateException("policies are in readonly mode");
+ }
+ return pulsarResources.getNamespaceResources()
+ .setPoliciesAsync(targetTopicName.getNamespaceObject(), policies -> {
+ Map<String, Set<AuthAction>> targetRoles = policies.auth_policies
+ .getTopicAuthentication()
+ .computeIfAbsent(targetTopicName.toString(), __ -> new HashMap<>());
+ sourceRoles.forEach((role, actions) -> {
Review Comment:
Fixed in 369553fd03: the source permissions are now read in the update callback, from the policies it updates on every attempt, so a retry after a concurrent revocation doesn't restore it; the nothing-to-copy check reads the latest policies from the store instead of the cache. Added `testCopyPermissionsDoesNotRestoreAPermissionRevokedMeanwhile`, and the copy tests now run against a cache that hasn't caught up.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4343262168
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/ScalableTopicResources.java:
##########
@@ -287,6 +301,101 @@ public CompletableFuture<Void> createScalableTopicAsync(TopicName tn, ScalableTo
return getCache().create(topicPath(tn), metadata, PROPERTY_INDEX_EXTRACTOR);
}
+ /** The creation lock of a scalable topic, while {@link #runWithCreationLock} holds it. */
+ public interface CreationLock {
+ /**
+ * Complete normally if the lock is still held, or fail with {@link MetadataStoreException.LockBusyException}.
+ * An action checks it before each change the lock guards: a broker whose metadata session expired lost the
+ * lock, which another broker may have taken since.
+ */
+ CompletableFuture<Void> checkHeld();
+ }
+
+ /**
+ * Run {@code action} while holding the creation lock of the scalable topic {@code tn}.
+ *
+ * <p>Creating a scalable topic and migrating a regular topic to it take this lock, so they exclude each other,
+ * on every broker: a migration checks that the scalable topic doesn't exist, grants the regular topic's
+ * permissions on it and creates it, with no creation of the same topic in between that would get those
+ * permissions. A lock held by another creation or migration is retried for about a second, which covers a
+ * concurrent creation, as it holds the lock only while it writes the topic's metadata. Then this fails with
+ * {@link MetadataStoreException.LockBusyException}, without running {@code action}.
+ *
+ * <p>Each acquisition creates an ephemeral record of its own, named after the topic and a random token, and
+ * holds the lock if no other record of the topic exists then. So a broker that goes away doesn't keep the lock;
+ * releasing it deletes that record only, never the one of an acquisition made after this broker's metadata
+ * session expired; and a create that the store retried after its response was lost finds its own record.
+ */
+ public <T> CompletableFuture<T> runWithCreationLock(TopicName tn,
+ Function<CreationLock, CompletableFuture<T>> action) {
+ String lockDir = joinPath(CREATION_LOCK_PATH, tn.getNamespace());
+ String prefix = tn.getEncodedLocalName() + ":";
+ String name = prefix + UUID.randomUUID();
+ String path = joinPath(lockDir, name);
+ CreationLock lock = () -> getStore().get(path).thenAccept(record -> {
+ if (record.isEmpty() || !record.get().getStat().isCreatedBySelf()) {
+ throw new CompletionException(new MetadataStoreException.LockBusyException(
+ "Lost the creation lock of " + tn));
+ }
+ });
+ return acquireCreationLock(lockDir, prefix, name, 1).thenCompose(__ -> {
+ CompletableFuture<T> result = FutureUtil.supplySafely(() -> action.apply(lock));
+ return result.handle((___, ____) -> null)
+ .thenCompose(___ -> releaseCreationLock(path))
+ .thenCompose(___ -> result);
+ });
+ }
+
+ private CompletableFuture<Void> acquireCreationLock(String lockDir, String prefix, String name, int attempt) {
+ String path = joinPath(lockDir, name);
+ return getStore().put(path, new byte[0], Optional.of(-1L), Set.of(Option.Ephemeral.INSTANCE))
+ .<Void>thenApply(__ -> null)
+ .exceptionallyCompose(ex -> {
+ if (FutureUtil.unwrapCompletionException(ex)
+ instanceof MetadataStoreException.BadVersionException) {
+ // The record exists: the store retried a create of this acquisition whose response was
+ // lost. No other acquisition uses this name.
+ return CompletableFuture.completedFuture(null);
+ }
+ // Delete the record, in case it was created anyway.
+ return releaseCreationLock(path).thenCompose(__ -> CompletableFuture.failedFuture(ex));
+ })
+ // Read the store, not the cache, which may miss the record of a concurrent acquisition.
+ .thenCompose(__ -> getStore().getChildrenFromStore(lockDir))
Review Comment:
Fixed in 789881112f: once the acquisition's record may exist, any failure of the attempt (the create, or listing the records) now deletes the record before failing with the original error; a successful acquisition keeps it until its action completes. `ScalableTopicCreationLockTest.testAcquisitionThatFailsToListTheRecords` fails the first listing: the action doesn't run, the error is the injected one, and the next acquisition takes the lock.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4300318337
##########
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/ScalableTopicsImpl.java:
##########
@@ -190,6 +192,53 @@ public CompletableFuture<Void> removeAutoScalePolicyAsync(String topic) {
return asyncDeleteRequest(topicPath(tn).path("autoScalePolicy"));
}
+ // --- Permissions ---
+
+ @Override
+ public Map<String, Set<AuthAction>> getPermissions(String topic) throws PulsarAdminException {
+ return sync(() -> getPermissionsAsync(topic));
+ }
+
+ @Override
+ public CompletableFuture<Map<String, Set<AuthAction>>> getPermissionsAsync(String topic) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncGetRequest(topicPath(tn).path("permissions"),
+ new GenericType<Map<String, Set<AuthAction>>>() {});
+ });
+ }
+
+ @Override
+ public void grantPermission(String topic, String role, Set<AuthAction> actions) throws PulsarAdminException {
+ sync(() -> grantPermissionAsync(topic, role, actions));
+ }
+
+ @Override
+ public CompletableFuture<Void> grantPermissionAsync(String topic, String role, Set<AuthAction> actions) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncPostRequest(permissionsPath(tn, role), Entity.entity(actions, MediaType.APPLICATION_JSON));
+ });
+ }
+
+ @Override
+ public void revokePermissions(String topic, String role) throws PulsarAdminException {
+ sync(() -> revokePermissionsAsync(topic, role));
+ }
+
+ @Override
+ public CompletableFuture<Void> revokePermissionsAsync(String topic, String role) {
+ return FutureUtil.supplySafely(() -> {
+ TopicName tn = validateTopic(topic);
+ return asyncDeleteRequest(permissionsPath(tn, role));
+ });
+ }
+
+ /** The role is a single path component, even if it contains a '/', as in the regular-topic permission API. */
+ private WebTarget permissionsPath(TopicName tn, String role) {
+ return WebTargets.addParts(topicPath(tn).path("permissions"), new String[] {role});
Review Comment:
**[P2] Encode the topic local name as one path component too**
The role is encoded, but `topicPath(tn)` appends `tn.getLocalName()` directly with `WebTarget.path`. Scalable topic names permit `/`, so `topic://tenant/ns/a/b` produces `/admin/v2/scalable/tenant/ns/a/b/permissions/app`, which misses the single-component `{topic}` route. Get and revoke have the same problem, as do their CLI commands.
This also breaks supported pregrants with `allowAclChangesOnNonExistentTopics=true`, without needing the topic to exist. Please encode the local topic name as one component in all three permission requests and add reserved-character topic-name coverage alongside the existing role-name cases. This issue is in the changes stacked from #26949.
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java:
##########
@@ -840,6 +847,53 @@ public CompletableFuture<Void> removePermissionsAsync(TopicName topicName) {
});
}
+ @Override
+ public CompletableFuture<Void> copyPermissionsAsync(TopicName sourceTopicName, TopicName targetTopicName) {
+ return pulsarResources.getNamespaceResources().getPoliciesAsync(sourceTopicName.getNamespaceObject())
+ .thenCompose(sourcePolicies -> {
+ Map<String, Set<AuthAction>> sourceRoles = sourcePolicies
+ .map(policies -> policies.auth_policies.getTopicAuthentication()
+ .get(sourceTopicName.toString()))
+ .orElse(null);
+ if (sourceRoles == null || sourceRoles.isEmpty()) {
+ // Nothing to copy: don't require the policies to be writable.
+ return CompletableFuture.completedFuture(null);
+ }
+ return getPoliciesReadOnlyAsync().thenCompose(readonly -> {
+ if (readonly) {
+ log.debug("Policies are read-only. Broker cannot do read-write operations");
+ throw new IllegalStateException("policies are in readonly mode");
+ }
+ return pulsarResources.getNamespaceResources()
+ .setPoliciesAsync(targetTopicName.getNamespaceObject(), policies -> {
+ Map<String, Set<AuthAction>> targetRoles = policies.auth_policies
+ .getTopicAuthentication()
+ .computeIfAbsent(targetTopicName.toString(), __ -> new HashMap<>());
+ sourceRoles.forEach((role, actions) -> {
Review Comment:
**[P1] Read source grants from the policy version being updated**
Another broker can complete a source-topic revocation while this broker still caches the old policy. `sourceRoles` captures that old grant; although the namespace CAS refreshes and retries against current policies, this closure keeps merging the old map, permanently restoring the revoked role's access on `topic://`.
Source and target are in the same namespace for migration, so derive the source grants from the `policies` supplied to the update callback on every attempt. A deterministic local test returning an old policy from the initial lookup and a post-revocation policy to the update callback reproduces the restored grant. Please add refresh/CAS-retry regression coverage.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/ScalableTopics.java:
##########
@@ -344,7 +348,14 @@ private CompletableFuture<Void> doMigrateToScalableAsync(TopicName scalableName,
ScalableTopicController.createMigratedMetadata(persistentBase, partitions,
pulsar().getConfiguration().getScalableTopicEntryBucketBudget(),
pulsar().getConfiguration().getScalableTopicEntryBucketMaxPerSegment());
+ // The new segments are authorized with the permissions of the scalable topic, a distinct
+ // identity, while the legacy segments keep those of the persistent:// topics they wrap. So
+ // grant the regular topic's permissions on the scalable topic before the commit: the V5
+ // clients that follow the layout change are then authorized on the new segments as soon as
+ // they see them. Namespace permissions apply to both topics already. Those of a single
+ // partition are not copied: the new segments mix its data with the other partitions' data.
return createMigratedChildTopicsAsync(scalableName, metadata)
+ .thenCompose(__ -> getAuthorizationService().copyPermissionsAsync(persistentBase, scalableName))
.thenCompose(__ -> resources().createScalableTopicAsync(scalableName, metadata))
Review Comment:
**[P1] Reserve the target before copying its permissions**
The initial absence check does not reserve `scalableName`. A concurrent native `createScalableTopic` can win after that check; this step then copies the source grants onto that live, independently created topic before the following metadata creation fails with 409. Those grants remain even though migration failed, and migration requires only produce permission on the source, not permission to administer the target.
Segment creation does not prevent this interleaving: a non-partitioned migration creates child ID 1, while native creation with one segment creates ID 0. Please coordinate native creation and migration for the same target across brokers before mutating ACLs, while keeping the migrated layout unpublished until permissions are ready. Add a test that pauses migration before the copy, creates the native target, and verifies that the losing migration does not change its grants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26953:
URL: https://github.com/apache/pulsar/pull/26953#discussion_r4322311106
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/ScalableTopics.java:
##########
@@ -344,7 +348,14 @@ private CompletableFuture<Void> doMigrateToScalableAsync(TopicName scalableName,
ScalableTopicController.createMigratedMetadata(persistentBase, partitions,
pulsar().getConfiguration().getScalableTopicEntryBucketBudget(),
pulsar().getConfiguration().getScalableTopicEntryBucketMaxPerSegment());
+ // The new segments are authorized with the permissions of the scalable topic, a distinct
+ // identity, while the legacy segments keep those of the persistent:// topics they wrap. So
+ // grant the regular topic's permissions on the scalable topic before the commit: the V5
+ // clients that follow the layout change are then authorized on the new segments as soon as
+ // they see them. Namespace permissions apply to both topics already. Those of a single
+ // partition are not copied: the new segments mix its data with the other partitions' data.
return createMigratedChildTopicsAsync(scalableName, metadata)
+ .thenCompose(__ -> getAuthorizationService().copyPermissionsAsync(persistentBase, scalableName))
.thenCompose(__ -> resources().createScalableTopicAsync(scalableName, metadata))
Review Comment:
Fixed in fa8791deba: creating a scalable topic (admin API and auto-creation on lookup) and migrating to it now take a per-topic ephemeral creation lock (`/topic-creation-locks/...`, outside the topic's path), and the migration holds it from its existence check, now read from the store rather than the cache, through the copy and the metadata commit, so no creation can interleave; a creation that finds it taken retries for about a second, then gets 409. `testCreationIsRefusedWhileAMigrationCopiesPermissions` pauses the migration at the copy and checks that a native create is refused and that the migration then completes; `testMigrationToAnExistingScalableTopicGrantsNothingOnIt` checks that a losing migration leaves the existing topic's permissions unchanged. (Not `LockManager`: two acquisitions from the same metadata session share or take over its lock.)
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org