You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "void-ptr974 (via GitHub)" <gi...@apache.org> on 2026/10/01 14:48:31 UTC

Re: [PR] [fix][broker] Prevent BookkeeperSchemaStorage, BookkeeperBucketSnapshotStorage, and compactors from bypassing namespace bookie affinity [pulsar]

void-ptr974 commented on code in PR #26705:
URL: https://github.com/apache/pulsar/pull/26705#discussion_r4156775400


##########
pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorTool.java:
##########
@@ -199,8 +252,15 @@ public static void main(String[] args) throws Exception {
         EventLoopGroup eventLoopGroup = EventLoopUtil.newEventLoopGroup(1, false,
                 new DefaultThreadFactory("compactor-io"));
 
+        LocalPoliciesResources localPoliciesResources = new LocalPoliciesResources(
+                store, brokerConfig.getMetadataStoreOperationTimeoutSeconds());
+        Optional<LocalPolicies> localPolicies =
+                localPoliciesResources.getLocalPolicies(topicName.getNamespaceObject());
+        BookKeeperClientContext bookKeeperClientContext = createBookKeeperClientContext(

Review Comment:
   Agreed. #26704 (or an equivalent initialization fix) is required for the first-ledger isolation guarantee. This PR can be reviewed independently, but I consider that fix a merge dependency: otherwise a newly created client can still place an auxiliary ledger before the initial /bookies mapping is available.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java:
##########
@@ -641,11 +645,48 @@ private CompletableFuture<Long> addEntry(LedgerHandle ledgerHandle, SchemaEntry
     }
 
     @NonNull
-    private CompletableFuture<LedgerHandle> createLedger(String schemaId) {
-        Map<String, byte[]> metadata = LedgerMetadataUtils.buildMetadataForSchema(schemaId);
+    @VisibleForTesting
+    CompletableFuture<LedgerHandle> createLedger(String schemaId) {
+        return CompletableFuture.completedFuture(schemaId)
+                .thenCompose(id -> {
+                    Optional<TopicName> topicName = getOwnerTopicName(id);
+                    if (topicName.isEmpty()) {
+                        return createLedger(id, bookKeeper, LedgerMetadataUtils.buildMetadataForSchema(id));
+                    }
+                    return pulsar.getBookKeeperClientContext(topicName.get())
+                            .thenCompose(clientContext -> createLedger(id, clientContext));

Review Comment:
   Fixed in 6498854. addNewSchemaEntryToStore now composes closeAsync() before returning the position, so the locator is published only after the ledger is closed. If close fails, it does not publish the locator and makes a best-effort attempt to delete the orphan ledger. I added a deterministic test for both close outcomes.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##########
@@ -1659,6 +1663,40 @@ public BookKeeper getBookKeeperClient() {
         }
     }
 
+    /**
+     * Resolve the placement policy for a topic and borrow the corresponding shared BookKeeper client.
+     *
+     * @param topicName the topic that owns the ledger
+     * @return a future that completes with the BookKeeper client and matching placement metadata
+     */
+    public CompletableFuture<BookKeeperClientContext> getBookKeeperClientContext(TopicName topicName) {
+        return CompletableFuture.completedFuture(topicName)
+                .thenCompose(name -> {
+                    Objects.requireNonNull(name, "topicName");
+                    return getPulsarResources().getLocalPolicies()
+                            .getLocalPoliciesAsync(name.getNamespaceObject());
+                }).thenCompose(localPolicies -> {
+                    EnsemblePlacementPolicyConfig placementPolicyConfig =
+                            BookKeeperPlacementPolicyConfigResolver.resolve(config, topicName, localPolicies)
+                                    .orElse(null);
+                    ManagedLedgerStorageClass defaultStorageClass =
+                            getManagedLedgerStorage().getDefaultStorageClass();
+                    if (!(defaultStorageClass instanceof BookkeeperManagedLedgerStorageClass bkStorageClass)) {

Review Comment:
   Fixed in 6498854. When there is no effective custom placement policy, the new overload uses the caller's existing BookKeeper client; schema and bucket pass the clients they already own. A custom policy still requires a BookkeeperManagedLedgerStorageClass so that we fail rather than silently ignore it. I added tests for the no-policy/non-BK-default case and both custom-policy cases.



##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##########
@@ -1659,6 +1663,40 @@ public BookKeeper getBookKeeperClient() {
         }
     }
 
+    /**
+     * Resolve the placement policy for a topic and borrow the corresponding shared BookKeeper client.
+     *
+     * @param topicName the topic that owns the ledger
+     * @return a future that completes with the BookKeeper client and matching placement metadata
+     */
+    public CompletableFuture<BookKeeperClientContext> getBookKeeperClientContext(TopicName topicName) {
+        return CompletableFuture.completedFuture(topicName)
+                .thenCompose(name -> {
+                    Objects.requireNonNull(name, "topicName");
+                    return getPulsarResources().getLocalPolicies()
+                            .getLocalPoliciesAsync(name.getNamespaceObject());
+                }).thenCompose(localPolicies -> {
+                    EnsemblePlacementPolicyConfig placementPolicyConfig =
+                            BookKeeperPlacementPolicyConfigResolver.resolve(config, topicName, localPolicies)
+                                    .orElse(null);
+                    ManagedLedgerStorageClass defaultStorageClass =
+                            getManagedLedgerStorage().getDefaultStorageClass();

Review Comment:
   The intended contract is to apply namespace affinity within the existing BookKeeper auxiliary-ledger backend. This PR does not route auxiliary ledgers by the topic's managed-ledger storage class. With no custom policy, schema and bucket retain their existing clients; with a custom policy, the context borrows from the default BookKeeper storage class. Following the topic's actual storage class would also require persisting backend identity with each auxiliary-ledger reference and routing reads and deletes accordingly. I documented this boundary in the getBookKeeperClientContext Javadoc in 6498854.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org