You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by xy...@apache.org on 2026/06/08 06:25:22 UTC

(pulsar) branch branch-4.2 updated (b07e36239f1 -> 712372f9ee1)

This is an automated email from the ASF dual-hosted git repository.

BewareMyPower pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


    from b07e36239f1 [improve][misc] Upgrade Jetty to 12.1.10 (#25943)
     new adc4fe4d3d1 [feat][pip] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25547)
     new 712372f9ee1 [feat][broker] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25707)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pip/pip-469.md                                     | 299 +++++++++++++++++++++
 .../apache/pulsar/broker/ServiceConfiguration.java |  12 +-
 .../org/apache/pulsar/broker/PulsarService.java    |  10 +-
 .../pulsar/broker/service/AbstractTopic.java       |   2 +-
 .../service/LegacyAwareTopicPoliciesService.java   | 142 ++++++++++
 .../service/MetadataStoreTopicPoliciesService.java | 277 +++++++++++++++++++
 .../SystemTopicBasedTopicPoliciesService.java      |   2 +-
 .../broker/service/TopicPoliciesService.java       |  15 +-
 .../broker/service/persistent/PersistentTopic.java |   8 +-
 .../admin/MetadataStoreTopicPoliciesTest.java      |  72 +++++
 .../pulsar/broker/admin/TopicPoliciesTest.java     | 112 ++++++--
 .../LegacyAwareTopicPoliciesServiceTest.java       | 190 +++++++++++++
 .../SystemTopicBasedTopicPoliciesServiceTest.java  |   4 +-
 .../broker/service/TopicPolicyTestUtils.java       |   7 +
 14 files changed, 1112 insertions(+), 40 deletions(-)
 create mode 100644 pip/pip-469.md
 create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesService.java
 create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/service/MetadataStoreTopicPoliciesService.java
 create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/MetadataStoreTopicPoliciesTest.java
 create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesServiceTest.java


(pulsar) 02/02: [feat][broker] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25707)

Posted by xy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

BewareMyPower pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 712372f9ee18ae5d5bd8e35ca6ef45473c0a3bb5
Author: Yunze Xu <xy...@163.com>
AuthorDate: Mon May 18 19:16:47 2026 +0800

    [feat][broker] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25707)
    
    (cherry picked from commit 8652efa4d60b791a5f1ee4e52f7ffda6ebbbb256)
---
 pip/pip-469.md                                     |  37 ++-
 .../apache/pulsar/broker/ServiceConfiguration.java |  12 +-
 .../org/apache/pulsar/broker/PulsarService.java    |  10 +-
 .../pulsar/broker/service/AbstractTopic.java       |   2 +-
 .../service/LegacyAwareTopicPoliciesService.java   | 142 +++++++++++
 .../service/MetadataStoreTopicPoliciesService.java | 277 +++++++++++++++++++++
 .../SystemTopicBasedTopicPoliciesService.java      |   2 +-
 .../broker/service/TopicPoliciesService.java       |  15 +-
 .../broker/service/persistent/PersistentTopic.java |   8 +-
 .../admin/MetadataStoreTopicPoliciesTest.java      |  72 ++++++
 .../pulsar/broker/admin/TopicPoliciesTest.java     | 112 +++++++--
 .../LegacyAwareTopicPoliciesServiceTest.java       | 190 ++++++++++++++
 .../SystemTopicBasedTopicPoliciesServiceTest.java  |   4 +-
 .../broker/service/TopicPolicyTestUtils.java       |   7 +
 14 files changed, 846 insertions(+), 44 deletions(-)

diff --git a/pip/pip-469.md b/pip/pip-469.md
index 69036381f0f..4734adc127d 100644
--- a/pip/pip-469.md
+++ b/pip/pip-469.md
@@ -105,8 +105,15 @@ Broker startup validates both backends:
 - `SystemTopicBasedTopicPoliciesService` must be instantiable.
 - The configured `topicPoliciesServiceClassName` must be instantiable.
 
-If either backend cannot be instantiated or started, broker startup fails. There is no per-request fallback from one
-backend to another.
+`LegacyAwareTopicPoliciesService#start` starts only the configured backend. It intentionally does not call
+`SystemTopicBasedTopicPoliciesService#start`, because that start path registers a namespace-bundle ownership listener
+whose only purpose is to eagerly create a reader on `<namespace>/__change_events` when a namespace bundle is loaded.
+Under legacy-aware routing, that eager optimization would be counterproductive because it can create readers for
+namespaces that do not have topic policies in `__change_events`. For legacy namespaces, the system-topic reader and
+policy cache are initialized lazily by the routed system-topic backend operations.
+
+If either backend cannot be instantiated, or if the configured backend cannot be started, broker startup fails. There is
+no per-request fallback from one backend to another.
 
 ### Namespace-scoped service routing
 
@@ -118,6 +125,10 @@ backend to another.
   the system-topic backend when the system topic exists.
 - Routing the same operations to the configured backend when the system topic does not exist.
 
+Listener registration is routed through `TopicPoliciesService#registerListenerAsync`. This lets the wrapper resolve the
+namespace backend before registering the listener, and the listener is registered only on the selected backend instead
+of being registered on both backends.
+
 The system-topic existence check can be cached per namespace in memory, but the routing rule is defined by actual topic
 existence rather than by new namespace metadata.
 
@@ -137,9 +148,13 @@ meaning the system-topic-backed topic-policies state is gone.
 
 - Topic names are normalized to the partitioned topic name, so all partitions share the same topic-policies record.
 - Global policies are stored in the configuration metadata store path:
-  `/admin/topic-policies/{tenant}/{namespace}/{domain}/{encodedTopic}`.
+  `/admin/topic-policies/global/{tenant}/{namespace}/{domain}/{encodedTopic}`.
 - Local policies are stored in the local metadata store path:
-  `/admin/local-policies/topic-policies/{tenant}/{namespace}/{domain}/{encodedTopic}`.
+  `/admin/topic-policies/local/{tenant}/{namespace}/{domain}/{encodedTopic}`.
+
+To avoid possible conflicts like the listener registered on the `/admin/local-policies` path from
+`BrokerService#handleMetadataChanges`, these two paths share the same root path `/admin/topic-policies`, which is not
+used by any other component.
 
 Each node stores a serialized `TopicPolicies` document. The backend writes and reads the two scopes independently:
 
@@ -159,6 +174,11 @@ managed-ledger metadata updates.
 
 ### Listener behavior
 
+`TopicPoliciesService` adds `registerListenerAsync(TopicName, TopicPolicyListener)` for listener registration. The
+existing synchronous `registerListener(TopicName, TopicPolicyListener)` method is retained as a deprecated compatibility
+hook for existing custom implementations, and the default async method delegates to it. Implementations that need async
+routing or initialization, such as `LegacyAwareTopicPoliciesService`, override `registerListenerAsync` directly.
+
 The backend registers watchers on both metadata stores:
 
 - A change on the local path re-reads the local node and notifies listeners with the latest local `TopicPolicies` or
@@ -173,6 +193,11 @@ append-only replay log; it relies on metadata-store notifications and read-after
 
 ### Public API
 
+The `TopicPoliciesService` extension point gains a default
+`CompletableFuture<Boolean> registerListenerAsync(TopicName, TopicPolicyListener)` method. Existing implementations
+remain compatible because `registerListener(TopicName, TopicPolicyListener)` is retained and used by the default async
+implementation.
+
 No new namespace policy field is introduced.
 
 No new namespace admin REST endpoint or Java admin client method is introduced.
@@ -221,6 +246,10 @@ This upgrade rule is intentionally conservative:
 This means some namespaces with an empty but already-created `__change_events` topic may continue using the
 system-topic backend. That is acceptable because it avoids missing legacy state.
 
+Existing custom `TopicPoliciesService` implementations that only implement the synchronous `registerListener` method
+continue to work through the default `registerListenerAsync` bridge. Implementations can override
+`registerListenerAsync` when registration itself needs asynchronous backend resolution or initialization.
+
 ## Downgrade / Rollback
 
 Rolling back to a broker version that does not understand legacy-aware routing returns topic-policies backend
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 0931aa64114..bf6ef7730e5 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1732,8 +1732,16 @@ public class ServiceConfiguration implements PulsarConfiguration {
 
     @FieldContext(
             category = CATEGORY_SERVER,
-            doc = "The class name of the topic policies service. The default config only takes affect when the "
-                    + "systemTopicEnable config is true"
+            doc = """
+                    The class name of the topic policies service. There are 2 built-in implementations:
+                    1. "org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService" (default)
+                      It stores a topic's policies in the `__change_events` topic. If `systemTopicEnabled` is false,
+                      the topic policies will just be disabled
+                    2. "org.apache.pulsar.broker.service.MetadataStoreTopicPoliciesService"
+                      It stores a topic's policies in the metadata store. If `systemTopicEnabled` is true and the
+                      topic's namespace has a `__change_events` topic, the policies will still be stored in the
+                      `__change_events` topic for backward compatibility.
+                    """
     )
     private String topicPoliciesServiceClassName =
             "org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService";
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 749dfc13b17..bf1f0e24848 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -106,6 +106,7 @@ import org.apache.pulsar.broker.resources.PulsarResources;
 import org.apache.pulsar.broker.rest.Topics;
 import org.apache.pulsar.broker.service.BrokerService;
 import org.apache.pulsar.broker.service.HealthChecker;
+import org.apache.pulsar.broker.service.LegacyAwareTopicPoliciesService;
 import org.apache.pulsar.broker.service.PulsarMetadataEventSynchronizer;
 import org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService;
 import org.apache.pulsar.broker.service.Topic;
@@ -2256,8 +2257,15 @@ public class PulsarService implements AutoCloseable, ShutdownService {
                 return TopicPoliciesService.DISABLED;
             }
         }
-        return (TopicPoliciesService) Reflections.createInstance(className,
+        final var configuredService = (TopicPoliciesService) Reflections.createInstance(className,
                 Thread.currentThread().getContextClassLoader());
+        if (!config.isSystemTopicEnabled()) {
+            LOG.info("[{}] System topic is disabled, using configured topic policies service without legacy routing",
+                    className);
+            return configuredService;
+        }
+        return new LegacyAwareTopicPoliciesService(this, new SystemTopicBasedTopicPoliciesService(this),
+                configuredService);
     }
 
     /**
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 8be1002b3a8..9f6f2e2ae38 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -552,7 +552,7 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
 
     protected void registerTopicPolicyListener() {
         brokerService.getPulsar().getTopicPoliciesService()
-                .registerListener(TopicName.getPartitionedTopicName(topic), this);
+                .registerListenerAsync(TopicName.getPartitionedTopicName(topic), this);
     }
 
     protected void unregisterTopicPolicyListener() {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesService.java
new file mode 100644
index 00000000000..6e1bfd7bacc
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesService.java
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import com.github.benmanes.caffeine.cache.AsyncCacheLoader;
+import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.google.common.annotations.VisibleForTesting;
+import java.time.Duration;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.Consumer;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.systopic.NamespaceEventsSystemTopicFactory;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.TopicPolicies;
+import org.jspecify.annotations.NonNull;
+
+/**
+ * Routes topic policy operations to the legacy system-topic backend when a namespace already has
+ * a topic-policy {@code __change_events} system topic, and otherwise to the configured backend.
+ */
+public class LegacyAwareTopicPoliciesService implements TopicPoliciesService {
+
+    private final AsyncLoadingCache<NamespaceName, Boolean> isLegacyNamespace;
+    @VisibleForTesting
+    final SystemTopicBasedTopicPoliciesService systemTopicService;
+    private final TopicPoliciesService configuredService;
+
+    public LegacyAwareTopicPoliciesService(PulsarService pulsar,
+                                           SystemTopicBasedTopicPoliciesService systemTopicService,
+                                           TopicPoliciesService configuredService) {
+        // Generally, we only need to check if the __change_events topic exists once because the __change_events topic
+        // should only be created by broker before the upgrade, where `SystemTopicBasedTopicPoliciesService` is
+        // configured as the topic policies service.
+        this.isLegacyNamespace = Caffeine.newBuilder().expireAfterWrite(Duration.ofHours(1))
+                .buildAsync(new AsyncCacheLoader<>() {
+                    @NonNull
+                    @Override
+                    public CompletableFuture<? extends Boolean> asyncLoad(NamespaceName key,
+                                                                          @NonNull Executor executor) {
+                        return NamespaceEventsSystemTopicFactory.checkSystemTopicExists(key, EventType.TOPIC_POLICY,
+                                pulsar);
+                    }
+                });
+        this.systemTopicService = systemTopicService;
+        this.configuredService = configuredService;
+        if (configuredService instanceof SystemTopicBasedTopicPoliciesService) {
+            throw new IllegalArgumentException(
+                    "configuredService should not be an instance of SystemTopicBasedTopicPoliciesService");
+        }
+    }
+
+    @Override
+    public void start(PulsarService pulsarService) {
+        // We should not call `systemTopicService.start()`, which just registers a namespace bundle listener to create
+        // a reader on `<namespace>/__change_events` when the namespace's bundle is loaded firstly. It's just an
+        // optimization to create the reader before loading any topic. However, it could create a reader on a namespace
+        // that does not even have the __change_events topic.
+        configuredService.start(pulsarService);
+    }
+
+    @Override
+    public void close() throws Exception {
+        try {
+            configuredService.close();
+        } finally {
+            systemTopicService.close();
+        }
+    }
+
+    @Override
+    public CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsync(TopicName topicName, GetType type) {
+        return resolveService(topicName.getNamespaceObject())
+                .thenCompose(service -> service.getTopicPoliciesAsync(topicName, type));
+    }
+
+    @Override
+    public CompletableFuture<Void> updateTopicPoliciesAsync(TopicName topicName, boolean isGlobalPolicy,
+                                                            boolean skipUpdateWhenTopicPolicyDoesntExist,
+                                                            Consumer<TopicPolicies> policyUpdater) {
+        return resolveService(topicName.getNamespaceObject())
+                .thenCompose(service -> service.updateTopicPoliciesAsync(topicName, isGlobalPolicy,
+                        skipUpdateWhenTopicPolicyDoesntExist, policyUpdater));
+    }
+
+    @Override
+    public CompletableFuture<Void> deleteTopicPoliciesAsync(TopicName topicName) {
+        return resolveService(topicName.getNamespaceObject())
+                .thenCompose(service -> service.deleteTopicPoliciesAsync(topicName));
+    }
+
+    @Override
+    public CompletableFuture<Void> deleteTopicPoliciesAsync(TopicName topicName,
+                                                            boolean keepGlobalPoliciesAfterDeleting) {
+        return resolveService(topicName.getNamespaceObject())
+                .thenCompose(service -> service.deleteTopicPoliciesAsync(topicName,
+                        keepGlobalPoliciesAfterDeleting));
+    }
+
+    @Override
+    public CompletableFuture<Boolean> registerListenerAsync(TopicName topicName, TopicPolicyListener listener) {
+        return resolveService(topicName.getNamespaceObject())
+                .thenCompose(service -> service.registerListenerAsync(topicName, listener));
+    }
+
+    @Override
+    public boolean registerListener(TopicName topicName, TopicPolicyListener listener) {
+        throw new RuntimeException("should not be called");
+    }
+
+    @Override
+    public void unregisterListener(TopicName topicName, TopicPolicyListener listener) {
+        configuredService.unregisterListener(topicName, listener);
+        systemTopicService.unregisterListener(topicName, listener);
+    }
+
+    @VisibleForTesting
+    CompletableFuture<TopicPoliciesService> resolveService(NamespaceName namespace) {
+        return isLegacyNamespace.get(namespace)
+                .thenApply(isLegacy -> isLegacy ? systemTopicService : configuredService);
+    }
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/MetadataStoreTopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/MetadataStoreTopicPoliciesService.java
new file mode 100644
index 00000000000..218c37472a3
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/MetadataStoreTopicPoliciesService.java
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.namespace.NamespaceService;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.TopicPolicies;
+import org.apache.pulsar.common.util.Codec;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.metadata.api.MetadataCache;
+import org.apache.pulsar.metadata.api.MetadataStore;
+import org.apache.pulsar.metadata.api.MetadataStoreException.NotFoundException;
+import org.apache.pulsar.metadata.api.Notification;
+import org.apache.pulsar.metadata.api.NotificationType;
+import org.jspecify.annotations.Nullable;
+
+/**
+ * Topic policies service backed by Pulsar metadata stores.
+ */
+@Slf4j
+public class MetadataStoreTopicPoliciesService implements TopicPoliciesService {
+
+    public static final String GLOBAL_POLICIES_ROOT = "/admin/topic-policies/global";
+    public static final String LOCAL_POLICIES_ROOT = "/admin/topic-policies/local";
+
+    private final AtomicBoolean closed = new AtomicBoolean(false);
+    private final Map<TopicName, List<TopicPolicyListener>> listeners = new ConcurrentHashMap<>();
+    private MetadataCache<TopicPolicies> localPoliciesCache;
+    private MetadataCache<TopicPolicies> globalPoliciesCache;
+
+    @Override
+    public void start(PulsarService pulsar) {
+        MetadataStore localStore = pulsar.getLocalMetadataStore();
+        MetadataStore configurationStore = pulsar.getConfigurationMetadataStore();
+        this.localPoliciesCache = localStore.getMetadataCache(TopicPolicies.class);
+        this.globalPoliciesCache = configurationStore.getMetadataCache(TopicPolicies.class);
+        localStore.registerListener(notification -> handleNotification(notification, false));
+        configurationStore.registerListener(notification -> handleNotification(notification, true));
+    }
+
+    @Override
+    public CompletableFuture<Void> deleteTopicPoliciesAsync(TopicName topicName) {
+        return deleteTopicPoliciesAsync(topicName, false);
+    }
+
+    @Override
+    public CompletableFuture<Void> deleteTopicPoliciesAsync(TopicName topicName,
+                                                            boolean keepGlobalPoliciesAfterDeleting) {
+        TopicName partitionedTopicName = normalizeTopicName(topicName);
+        if (NamespaceService.isHeartbeatNamespace(partitionedTopicName.getNamespaceObject())) {
+            return CompletableFuture.completedFuture(null);
+        }
+        if (closed.get()) {
+            return CompletableFuture.failedFuture(new BrokerServiceException(getClass().getName() + " is closed."));
+        }
+        CompletableFuture<Void> deleteLocal =
+                deleteIfExists(localPoliciesCache, pathFor(partitionedTopicName, false));
+        if (keepGlobalPoliciesAfterDeleting) {
+            return deleteLocal;
+        }
+        CompletableFuture<Void> deleteGlobal =
+                deleteIfExists(globalPoliciesCache, pathFor(partitionedTopicName, true));
+        return CompletableFuture.allOf(deleteLocal, deleteGlobal);
+    }
+
+    @Override
+    public CompletableFuture<Void> updateTopicPoliciesAsync(TopicName topicName, boolean isGlobalPolicy,
+                                                            boolean skipUpdateWhenTopicPolicyDoesntExist,
+                                                            Consumer<TopicPolicies> policyUpdater) {
+        TopicName partitionedTopicName = normalizeTopicName(topicName);
+        if (NamespaceService.isHeartbeatNamespace(partitionedTopicName.getNamespaceObject())) {
+            return CompletableFuture.failedFuture(new BrokerServiceException.NotAllowedException(
+                    "Not allowed to update topic policy for the heartbeat topic"));
+        }
+        if (closed.get()) {
+            return CompletableFuture.failedFuture(new BrokerServiceException(getClass().getName() + " is closed."));
+        }
+        MetadataCache<TopicPolicies> cache = cache(isGlobalPolicy);
+        String path = pathFor(partitionedTopicName, isGlobalPolicy);
+        CompletableFuture<TopicPolicies> updateFuture;
+        if (skipUpdateWhenTopicPolicyDoesntExist) {
+            updateFuture = cache.readModifyUpdate(path,
+                    current -> updatePolicies(Optional.of(current), isGlobalPolicy, policyUpdater));
+        } else {
+            updateFuture = cache.readModifyUpdateOrCreate(path,
+                    current -> updatePolicies(current, isGlobalPolicy, policyUpdater));
+        }
+        return updateFuture.thenAccept(__ -> { }).exceptionally(error -> {
+            if (skipUpdateWhenTopicPolicyDoesntExist
+                    && FutureUtil.unwrapCompletionException(error) instanceof NotFoundException) {
+                return null;
+            }
+            throw FutureUtil.wrapToCompletionException(error);
+        });
+    }
+
+    @Override
+    public CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsync(TopicName topicName, GetType type) {
+        TopicName partitionedTopicName = normalizeTopicName(topicName);
+        if (NamespaceService.isHeartbeatNamespace(partitionedTopicName.getNamespaceObject())) {
+            return CompletableFuture.completedFuture(Optional.empty());
+        }
+        if (closed.get()) {
+            return CompletableFuture.completedFuture(Optional.empty());
+        }
+        boolean global = type == GetType.GLOBAL_ONLY;
+        return cache(global).get(pathFor(partitionedTopicName, global))
+                .thenApply(policies -> policies.map(policy -> cloneWithScope(policy, global)));
+    }
+
+    @Override
+    public boolean registerListener(TopicName topicName, TopicPolicyListener listener) {
+        listeners.compute(normalizeTopicName(topicName), (__, topicListeners) -> {
+            if (topicListeners == null) {
+                topicListeners = new CopyOnWriteArrayList<>();
+            }
+            topicListeners.add(listener);
+            return topicListeners;
+        });
+        return true;
+    }
+
+    @Override
+    public void unregisterListener(TopicName topicName, TopicPolicyListener listener) {
+        listeners.computeIfPresent(normalizeTopicName(topicName), (__, topicListeners) -> {
+            topicListeners.remove(listener);
+            return topicListeners.isEmpty() ? null : topicListeners;
+        });
+    }
+
+    @Override
+    public void close() {
+        if (closed.compareAndSet(false, true)) {
+            listeners.clear();
+            if (localPoliciesCache != null) {
+                localPoliciesCache.invalidateAll();
+            }
+            if (globalPoliciesCache != null) {
+                globalPoliciesCache.invalidateAll();
+            }
+        }
+    }
+
+    private MetadataCache<TopicPolicies> cache(boolean isGlobalPolicy) {
+        return isGlobalPolicy ? globalPoliciesCache : localPoliciesCache;
+    }
+
+    private CompletableFuture<Void> deleteIfExists(MetadataCache<TopicPolicies> cache, String path) {
+        return cache.delete(path).handle((__, error) -> {
+            cache.invalidate(path);
+            if (error == null || FutureUtil.unwrapCompletionException(error) instanceof NotFoundException) {
+                return null;
+            }
+            throw FutureUtil.wrapToCompletionException(error);
+        });
+    }
+
+    private static TopicPolicies updatePolicies(Optional<TopicPolicies> currentPolicies,
+                                                boolean isGlobalPolicy,
+                                                Consumer<TopicPolicies> policyUpdater) {
+        TopicPolicies policies = currentPolicies.map(TopicPolicies::clone).orElseGet(TopicPolicies::new);
+        policies.setIsGlobal(isGlobalPolicy);
+        policyUpdater.accept(policies);
+        return policies;
+    }
+
+    private void handleNotification(Notification notification, boolean isGlobalPolicy) {
+        if (closed.get()
+                || (notification.getType() != NotificationType.Created
+                && notification.getType() != NotificationType.Modified
+                && notification.getType() != NotificationType.Deleted)) {
+            return;
+        }
+        String path = notification.getPath();
+        String root = isGlobalPolicy ? GLOBAL_POLICIES_ROOT : LOCAL_POLICIES_ROOT;
+        Optional<TopicName> topicName = topicNameFromPath(root, path);
+        if (topicName.isEmpty()) {
+            return;
+        }
+        MetadataCache<TopicPolicies> cache = cache(isGlobalPolicy);
+        cache.invalidate(path);
+        if (notification.getType() == NotificationType.Deleted) {
+            notifyListeners(topicName.get(), null);
+            return;
+        }
+        cache.get(path).whenComplete((policies, error) -> {
+            if (error != null) {
+                log.warn("[{}] Failed to refresh topic policies after metadata notification", path, error);
+                return;
+            }
+            notifyListeners(topicName.get(),
+                    policies.map(policy -> cloneWithScope(policy, isGlobalPolicy)).orElse(null));
+        });
+    }
+
+    private void notifyListeners(TopicName topicName, @Nullable TopicPolicies policies) {
+        List<TopicPolicyListener> topicListeners = listeners.get(topicName);
+        if (topicListeners == null) {
+            return;
+        }
+        for (TopicPolicyListener listener : topicListeners) {
+            try {
+                listener.onUpdate(policies == null ? null : policies.clone());
+            } catch (Throwable error) {
+                log.error("[{}] Call topic policy listener error", topicName, error);
+            }
+        }
+    }
+
+    private static TopicName normalizeTopicName(TopicName topicName) {
+        return TopicName.get(topicName.getPartitionedTopicName());
+    }
+
+    private static TopicPolicies cloneWithScope(TopicPolicies policies, boolean isGlobalPolicy) {
+        TopicPolicies cloned = policies.clone();
+        cloned.setIsGlobal(isGlobalPolicy);
+        return cloned;
+    }
+
+    @VisibleForTesting
+    public CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesDirectFromStore(TopicName topicName,
+                                                                                      boolean isGlobal) {
+        String path = pathFor(topicName, isGlobal);
+        MetadataCache<TopicPolicies> c = cache(isGlobal);
+        c.invalidate(path);
+        return c.get(path).thenApply(opt -> opt.map(p -> cloneWithScope(p, isGlobal)));
+    }
+
+    @VisibleForTesting
+    static String pathFor(TopicName topicName, boolean isGlobalPolicy) {
+        TopicName partitionedTopicName = normalizeTopicName(topicName);
+        return (isGlobalPolicy ? GLOBAL_POLICIES_ROOT : LOCAL_POLICIES_ROOT)
+                + "/" + partitionedTopicName.getTenant()
+                + "/" + partitionedTopicName.getNamespacePortion()
+                + "/" + partitionedTopicName.getDomain()
+                + "/" + partitionedTopicName.getEncodedLocalName();
+    }
+
+    @VisibleForTesting
+    private static Optional<TopicName> topicNameFromPath(String root, String path) {
+        if (!path.startsWith(root + "/")) {
+            return Optional.empty();
+        }
+        String[] parts = path.substring(root.length() + 1).split("/", 4);
+        if (parts.length != 4) {
+            return Optional.empty();
+        }
+        return Optional.of(TopicName.get(parts[2], parts[0], parts[1], Codec.decode(parts[3])));
+    }
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
index ad18af308e2..f8a5d593661 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
@@ -647,7 +647,7 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
         return systemTopicClient.newReaderAsync();
     }
 
-    private void removeOwnedNamespaceBundleAsync(NamespaceBundle namespaceBundle) {
+    void removeOwnedNamespaceBundleAsync(NamespaceBundle namespaceBundle) {
         NamespaceName namespace = namespaceBundle.getNamespaceObject();
         if (NamespaceService.isHeartbeatNamespace(namespace)) {
             return;
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPoliciesService.java
index 239c1d3d9ba..7f49686a17f 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPoliciesService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPoliciesService.java
@@ -100,6 +100,15 @@ public interface TopicPoliciesService extends AutoCloseable {
     default void close() throws Exception {
     }
 
+
+    /**
+     * @implNote This method is never called unless by the default implementation of
+     * {@link TopicPoliciesService#registerListenerAsync(TopicName, TopicPolicyListener)}, which is actually called
+     * internally. This method is only retained for backward compatibility on custom implementations.
+     */
+    @Deprecated
+    boolean registerListener(TopicName topicName, TopicPolicyListener listener);
+
     /**
      * Registers a listener for topic policies updates.
      *
@@ -110,10 +119,10 @@ public interface TopicPoliciesService extends AutoCloseable {
      * guaranteed to be received by the listener.
      * In summary, the listener is guaranteed to receive only the latest value.
      * </p>
-     *
-     * @return true if the listener is registered successfully
      */
-    boolean registerListener(TopicName topicName, TopicPolicyListener listener);
+    default CompletableFuture<Boolean> registerListenerAsync(TopicName topicName, TopicPolicyListener listener) {
+        return CompletableFuture.completedFuture(registerListener(topicName, listener));
+    }
 
     /**
      * Unregister the topic policies listener.
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index e7a006185af..89e1a47de30 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -4726,7 +4726,10 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
     protected CompletableFuture<Void> initTopicPolicy() {
         final var topicPoliciesService = brokerService.pulsar().getTopicPoliciesService();
         final var partitionedTopicName = TopicName.getPartitionedTopicName(topic);
-        if (topicPoliciesService.registerListener(partitionedTopicName, this)) {
+        return topicPoliciesService.registerListenerAsync(partitionedTopicName, this).thenCompose(registered -> {
+            if (!registered) {
+                return CompletableFuture.completedFuture(null);
+            }
             if (ExtensibleLoadManagerImpl.isInternalTopic(topic)) {
                 return CompletableFuture.completedFuture(null);
             }
@@ -4738,8 +4741,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
                     TopicPoliciesService.GetType.LOCAL_ONLY))
             .thenAcceptAsync(optionalPolicies -> optionalPolicies.ifPresent(this::onUpdate),
                             brokerService.getTopicOrderedExecutor());
-        }
-        return CompletableFuture.completedFuture(null);
+        });
     }
 
     @VisibleForTesting
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/MetadataStoreTopicPoliciesTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/MetadataStoreTopicPoliciesTest.java
new file mode 100644
index 00000000000..e7fefa16497
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/MetadataStoreTopicPoliciesTest.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.broker.admin;
+
+import org.apache.pulsar.broker.service.MetadataStoreTopicPoliciesService;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker-admin")
+public class MetadataStoreTopicPoliciesTest extends TopicPoliciesTest {
+
+    @BeforeClass(alwaysRun = true)
+    @Override
+    protected void setup() throws Exception {
+        conf.setTopicPoliciesServiceClassName(MetadataStoreTopicPoliciesService.class.getName());
+        super.setup();
+    }
+
+    @Override
+    protected void clearTopicPoliciesCache() {
+    }
+
+    @Test(enabled = false)
+    @Override
+    public void testTopicPolicyInitialValueWithNamespaceAlreadyLoaded() throws Exception {
+        // This test is specific to SystemTopicBasedTopicPoliciesService (uses getPoliciesCacheInit).
+        // Not applicable to MetadataStoreTopicPoliciesService.
+    }
+
+    @Test(enabled = false)
+    @Override
+    public void testSystemTopicShouldBeCompacted() throws Exception {
+        // Relies on __change_events system topic, which does not exist with MetadataStoreTopicPoliciesService.
+    }
+
+    @Test(enabled = false)
+    @Override
+    public void testPoliciesCanBeDeletedWithTopic() throws Exception {
+        // Directly accesses __change_events PersistentTopic for compaction.
+        // Not applicable to MetadataStoreTopicPoliciesService.
+    }
+
+    @Test(enabled = false)
+    @Override
+    public void testProduceChangesWithEncryptionRequired() throws Exception {
+        // Checks __change_events LAC, which does not exist with MetadataStoreTopicPoliciesService.
+    }
+
+    @Test(enabled = false)
+    @Override
+    public void testTopicPoliciesAfterCompaction(String reloadPolicyType) throws Exception {
+        // The "Recreate_Service" variant creates a new SystemTopicBasedTopicPoliciesService,
+        // which is not applicable to MetadataStoreTopicPoliciesService.
+    }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
index 47b0d63aba6..099b6ed8921 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
@@ -119,7 +119,9 @@ import org.glassfish.jersey.client.JerseyClient;
 import org.glassfish.jersey.client.JerseyClientBuilder;
 import org.mockito.Mockito;
 import org.testng.Assert;
+import org.testng.annotations.AfterClass;
 import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeClass;
 import org.testng.annotations.BeforeMethod;
 import org.testng.annotations.DataProvider;
 import org.testng.annotations.Test;
@@ -144,10 +146,11 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     private final int testTopicPartitions = 2;
 
-    @BeforeMethod
+    @BeforeClass(alwaysRun = true)
     @Override
     protected void setup() throws Exception {
         this.conf.setDefaultNumberOfNamespaceBundles(1);
+        this.conf.setForceDeleteNamespaceAllowed(true);
         super.internalSetup();
 
         admin.clusters().createCluster("test", ClusterData.builder().serviceUrl(pulsar.getWebServiceAddress()).build());
@@ -158,15 +161,48 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         admin.topics().createPartitionedTopic(testTopic, testTopicPartitions);
         Producer producer = pulsarClient.newProducer().topic(testTopic).create();
         producer.close();
-        waitForZooKeeperWatchers();
     }
 
-    @AfterMethod(alwaysRun = true)
+    @AfterClass(alwaysRun = true)
     @Override
     public void cleanup() throws Exception {
         super.internalCleanup();
     }
 
+    @BeforeMethod
+    void setupTestTopic() throws Exception {
+        // Recreate namespace to clear any policies set by previous tests
+        try {
+            admin.topics().deletePartitionedTopic(testTopic, true);
+        } catch (PulsarAdminException.NotFoundException e) {
+            // topic may already be deleted
+        }
+        try {
+            admin.namespaces().deleteNamespace(myNamespace, true);
+        } catch (PulsarAdminException.NotFoundException e) {
+            // namespace may already be deleted
+        }
+        try {
+            admin.namespaces().deleteNamespace(myNamespaceV1, true);
+        } catch (PulsarAdminException.NotFoundException e) {
+            // namespace may already be deleted
+        }
+        admin.namespaces().createNamespace(testTenant + "/" + testNamespace, Set.of("test"));
+        admin.namespaces().createNamespace(myNamespaceV1);
+        admin.topics().createPartitionedTopic(testTopic, testTopicPartitions);
+        // Acquire namespace bundle ownership so tests that call getOrCreateTopic() directly succeed.
+        // Without this, services that don't create a __change_events reader (e.g. MetadataStoreTopicPoliciesService)
+        // leave the bundle unowned after namespace recreation and the first broker-side topic load fails.
+        admin.lookups().lookupTopic(testTopic + "-partition-0");
+    }
+
+    @AfterMethod(alwaysRun = true)
+    void afterMethodCleanup() throws Exception{
+        admin.brokers().updateDynamicConfiguration("maxPublishRatePerTopicInMessages", "0");
+        admin.brokers().updateDynamicConfiguration("maxPublishRatePerTopicInBytes", "0");
+        clearTopicPoliciesCache();
+    }
+
     @Test
     public void updatePropertiesForAutoCreatedTopicTest() throws Exception {
         TopicName topicName = TopicName.get(
@@ -485,8 +521,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test(dataProvider = "clientRequestType")
     public void testPriorityOfGlobalPolicies(String clientRequestType) throws Exception {
-        final SystemTopicBasedTopicPoliciesService topicPoliciesService =
-                (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+        final TopicPoliciesService topicPoliciesService =
+                pulsar.getTopicPoliciesService();
         final JerseyClient httpClient = JerseyClientBuilder.createClient();
         // create topic and load it up.
         final String namespace = myNamespace;
@@ -566,8 +602,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test(dataProvider = "clientRequestType")
     public void testPriorityOfGlobalPolicies2(String clientRequestType) throws Exception {
-        final SystemTopicBasedTopicPoliciesService topicPoliciesService =
-                (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+        final TopicPoliciesService topicPoliciesService =
+                pulsar.getTopicPoliciesService();
         final JerseyClient httpClient = JerseyClientBuilder.createClient();
         // create topic and load it up.
         final String namespace = myNamespace;
@@ -653,8 +689,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         final TopicName topicName = TopicName.get(topic);
         admin.topics().createNonPartitionedTopic(topic);
         pulsarClient.newProducer().topic(topic).create().close();
-        final SystemTopicBasedTopicPoliciesService topicPoliciesService =
-                (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+        final TopicPoliciesService topicPoliciesService =
+                pulsar.getTopicPoliciesService();
 
         // Set non-global policy of the limitation of max consumers.
         // Set global policy of the limitation of max producers.
@@ -695,8 +731,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         final TopicName topicName = TopicName.get(topic);
         admin.topics().createNonPartitionedTopic(topic);
         pulsarClient.newProducer().topic(topic).create().close();
-        final SystemTopicBasedTopicPoliciesService topicPoliciesService =
-                (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+        final TopicPoliciesService topicPoliciesService =
+                pulsar.getTopicPoliciesService();
 
         // Set non-global policy of the limitation of max consumers.
         // Set global policy of the persistence policies.
@@ -2596,10 +2632,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test
     public void testPublishRateInDifferentLevelPolicy() throws Exception {
-        cleanup();
-        conf.setMaxPublishRatePerTopicInMessages(5);
-        conf.setMaxPublishRatePerTopicInBytes(50L);
-        setup();
+        admin.brokers().updateDynamicConfiguration("maxPublishRatePerTopicInMessages", "5");
+        admin.brokers().updateDynamicConfiguration("maxPublishRatePerTopicInBytes", "50");
 
         final String topicName = "persistent://" + myNamespace + "/test-" + UUID.randomUUID();
         pulsarClient.newProducer().topic(topicName).create().close();
@@ -2890,9 +2924,7 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test
     public void testMaxUnackedMessagesOnSubscriptionPriority() throws Exception {
-        cleanup();
-        conf.setMaxUnackedMessagesPerSubscription(30);
-        setup();
+        restartBroker(conf -> conf.setMaxUnackedMessagesPerSubscription(30));
         final String topic = "persistent://" + myNamespace + "/test-" + UUID.randomUUID();
         // init cache
         @Cleanup
@@ -2955,6 +2987,9 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
                 && admin.topicPolicies().getMaxUnackedMessagesOnSubscription(topic) == null);
         messages = getMsgReceived(consumer1, Integer.MAX_VALUE);
         assertEquals(messages.size(), defaultMaxUnackedMsgOnBroker);
+
+        // restore default config
+        restartBroker(conf -> conf.setMaxUnackedMessagesPerSubscription(4 * 50000));
     }
 
     private void produceMsg(Producer producer, int msgNum) throws Exception{
@@ -3139,14 +3174,16 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test(timeOut = 30000)
     public void testAutoCreationDisabled() throws Exception {
-        cleanup();
-        conf.setAllowAutoTopicCreation(false);
-        setup();
+        admin.brokers().updateDynamicConfiguration("allowAutoTopicCreation", "false");
+
         final String topic = testTopic + UUID.randomUUID();
         admin.topics().createPartitionedTopic(topic, 3);
         pulsarClient.newProducer().topic(topic).create().close();
         //should not fail
         assertNull(admin.topicPolicies().getMessageTTL(topic));
+
+        // restore default
+        admin.brokers().updateDynamicConfiguration("allowAutoTopicCreation", "true");
     }
 
     @Test
@@ -3270,6 +3307,12 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         pulsarClient.newConsumer().topic(topic)
                 .subscriptionType(SubscriptionType.Shared).subscriptionName("test")
                 .subscribe().close();
+
+        // restore dynamic broker config and conf object
+        pulsar.getConfiguration().setSubscriptionTypesEnabled(
+                Set.of("Exclusive", "Shared", "Failover", "Key_Shared"));
+        admin.brokers().updateDynamicConfiguration("subscriptionTypesEnabled",
+                "Exclusive,Shared,Failover,Key_Shared");
     }
 
     @Test(timeOut = 20000)
@@ -3600,7 +3643,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
     }
 
     @Test
-    public void testDoNotCreateSystemTopicForHeartbeatNamespace() {
+    public void testDoNotCreateSystemTopicForHeartbeatNamespace() throws Exception {
+        initEventsTopicAndPartitions();
         assertTrue(pulsar.getBrokerService().getTopics().size() > 0);
         pulsar.getBrokerService().getTopics().forEach((k, v) -> {
             TopicName topicName = TopicName.get(k);
@@ -3661,8 +3705,13 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
     }
 
     private void triggerAndWaitNewTopicCompaction(String topicName) throws Exception {
-        PersistentTopic tp =
-                (PersistentTopic) pulsar.getBrokerService().getTopic(topicName, false).join().get();
+        Optional<Topic> topicOpt =
+                pulsar.getBrokerService().getTopic(topicName, false).join();
+        if (topicOpt.isEmpty()) {
+            // Topic doesn't exist (e.g., when not using system-topic-based policies service), nothing to compact.
+            return;
+        }
+        PersistentTopic tp = (PersistentTopic) topicOpt.get();
         // Wait for the old task finish.
         Awaitility.await().untilAsserted(() -> {
             CompletableFuture<Long> compactionTask = WhiteboxImpl.getInternalState(tp, "currentCompaction");
@@ -3681,7 +3730,7 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
      * It is not a thread safety method, something will go to a wrong pointer if there is a task is trying to load a
      * topic policies.
      */
-    private void clearTopicPoliciesCache() {
+    protected void clearTopicPoliciesCache() {
         TopicPoliciesService topicPoliciesService = pulsar.getTopicPoliciesService();
         if (topicPoliciesService instanceof TopicPoliciesService.TopicPoliciesServiceDisabled) {
             return;
@@ -3911,8 +3960,8 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
                         .isNull());
         admin.topicPolicies(true).setRetention(topic, new RetentionPolicies(1,
                 2));
-        SystemTopicBasedTopicPoliciesService topicPoliciesService =
-                (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+        TopicPoliciesService topicPoliciesService =
+                pulsar.getTopicPoliciesService();
 
         // check global topic policies can be added correctly.
         Awaitility.await().untilAsserted(() -> assertNotNull(
@@ -3956,6 +4005,7 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test
     public void testMaxMessageSizeWithChunking() throws Exception {
+        final var maxMessageSize = this.conf.getMaxMessageSize();
         this.conf.setMaxMessageSize(1000);
 
         @Cleanup
@@ -3984,6 +4034,7 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
         // chunk message send success
         producer.send(new byte[2000]);
+        this.conf.setMaxMessageSize(maxMessageSize);
     }
 
     @Test(timeOut = 30000)
@@ -4037,6 +4088,7 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
 
     @Test
     public void testProduceChangesWithEncryptionRequired() throws Exception {
+        initEventsTopicAndPartitions();
         final String beforeLac = admin.topics().getInternalStats(topicPolicyEventsTopic).lastConfirmedEntry;
         admin.namespaces().setEncryptionRequiredStatus(myNamespace, true);
         // just an update to trigger writes on __change_events
@@ -4493,4 +4545,10 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         assertEquals(offloadPolicies.getManagedLedgerOffloadThresholdInBytes(), (Long) (1024 * 1024 * 10L),
                 "Should inherit offload threshold from legacy namespace policy");
     }
+
+    private void initEventsTopicAndPartitions() throws Exception {
+        try (Producer<?> producer = pulsarClient.newProducer().topic(testTopic).create()) {
+            // No-op. Creating the producer initializes the events topic and partitions.
+        }
+    }
 }
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesServiceTest.java
new file mode 100644
index 00000000000..47a7de0528d
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesServiceTest.java
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import java.time.Duration;
+import java.util.Optional;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.TopicPolicies;
+import org.awaitility.Awaitility;
+import org.awaitility.core.ThrowingRunnable;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+/**
+ * Test order: testUpgrade() -> other tests (with MetadataStoreTopicPoliciesService configured) -> testDowngrade().
+ */
+@Test(groups = "broker")
+public class LegacyAwareTopicPoliciesServiceTest extends MockedPulsarServiceBaseTest {
+
+    private static final String metaNamespace = "public/meta-ns";
+
+    @BeforeClass
+    @Override
+    protected void setup() throws Exception {
+        super.internalSetup();
+        super.setupDefaultTenantAndNamespace();
+    }
+
+    @AfterClass
+    @Override
+    protected void cleanup() throws Exception {
+        super.internalCleanup();
+    }
+
+    @Test(priority = -1)
+    public void testUpgrade() throws Exception {
+        final var topic = "test-upgrade";
+        admin.topics().createNonPartitionedTopic(topic);
+        admin.topicPolicies().setCompactionThreshold(topic, 100);
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies().getCompactionThreshold(topic), 100));
+
+        restartBroker(conf -> {
+            conf.setSystemTopicEnabled(false);
+            conf.setTopicPoliciesServiceClassName(MetadataStoreTopicPoliciesService.class.getName());
+        });
+        // The policies will be lost because when system topic is disabled, it will not try to read policies from the
+        // __change_events topic
+        assertNull(admin.topicPolicies().getCompactionThreshold(topic));
+
+        restartBroker(conf -> conf.setSystemTopicEnabled(true));
+        // The default namespace still read policies from the __change_events topic if it exists
+        assertEquals(admin.topicPolicies().getCompactionThreshold(topic), 100);
+        assertFalse(pulsar.getLocalMetadataStore().exists(MetadataStoreTopicPoliciesService.LOCAL_POLICIES_ROOT).get());
+
+        // The global policies are still stored in the __change_events topic
+        admin.topicPolicies(true).setCompactionThreshold(topic, 200);
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies(true).getCompactionThreshold(topic), 200));
+        assertFalse(pulsar.getConfigurationMetadataStore()
+                .exists(MetadataStoreTopicPoliciesService.GLOBAL_POLICIES_ROOT).get());
+
+        admin.topicPolicies().deleteTopicPolicies(topic);
+        waitUntilAssert(() -> assertNull(admin.topicPolicies().getCompactionThreshold(topic)));
+
+        admin.namespaces().createNamespace(metaNamespace);
+    }
+
+    @Test(priority = 1)
+    public void testDowngrade() throws Exception {
+        final var topic1 = "downgrade"; // in default namespace
+        admin.topics().createNonPartitionedTopic(topic1);
+        admin.topicPolicies().setCompactionThreshold(topic1, 1);
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies().getCompactionThreshold(topic1), 1));
+
+        final var topic2 = metaNamespace + "/downgrade";
+        admin.topics().createNonPartitionedTopic(topic2);
+        admin.topicPolicies().setCompactionThreshold(topic2, 2);
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies().getCompactionThreshold(topic2), 2));
+
+        restartBroker(conf ->
+                conf.setTopicPoliciesServiceClassName(SystemTopicBasedTopicPoliciesService.class.getName()));
+        assertEquals(admin.topicPolicies().getCompactionThreshold(topic1), 1);
+        // The policies will be lost because they are not stored in the __change_events topic
+        assertNull(admin.topicPolicies().getCompactionThreshold(topic2));
+    }
+
+    @DataProvider
+    public Object[][] namespaces() {
+        return new Object[][] {
+                { "public/default" },
+                { metaNamespace }
+        };
+    }
+
+    @Test(dataProvider = "namespaces")
+    public void testPoliciesOperations(String namespace) throws Exception {
+        final var topicName = TopicName.get(namespace + "/test-policies-operations");
+        final var topic = topicName.toString();
+        admin.topics().createNonPartitionedTopic(topic);
+
+        final var compactionThreshold = new AtomicLong(0);
+        // Verify the exception thrown from one listener does not affect other listeners
+        pulsar.getTopicPoliciesService().registerListenerAsync(topicName, __ -> {
+            throw new RuntimeException("injected failure");
+        }).get();
+        pulsar.getTopicPoliciesService().registerListenerAsync(topicName, policies ->
+                Optional.ofNullable(policies).map(TopicPolicies::getCompactionThreshold).ifPresentOrElse(
+                        compactionThreshold::set, () -> compactionThreshold.set(-1))).get();
+
+        // Verify Created events are handled
+        admin.topicPolicies(false).setCompactionThreshold(topic, 100);
+        waitUntilAssert(() -> assertEquals(compactionThreshold.get(), 100));
+        final var localStore = pulsar.getLocalMetadataStore();
+        final var configurationStore = pulsar.getConfigurationMetadataStore();
+
+        if (namespace.equals(metaNamespace)) {
+            assertTrue(localStore.exists(MetadataStoreTopicPoliciesService.pathFor(topicName, false)).get());
+            assertFalse(configurationStore.exists(MetadataStoreTopicPoliciesService.pathFor(topicName, true)).get());
+        }
+
+        admin.topicPolicies(true).setCompactionThreshold(topic, 200);
+        waitUntilAssert(() -> assertEquals(compactionThreshold.get(), 200));
+        if (namespace.equals(metaNamespace)) {
+            assertTrue(configurationStore.exists(MetadataStoreTopicPoliciesService.pathFor(topicName, true)).get());
+        }
+
+        // Verify Modified events are handled
+        admin.topicPolicies(false).setCompactionThreshold(topic, 300);
+        waitUntilAssert(() -> assertEquals(compactionThreshold.get(), 300));
+
+        admin.topicPolicies(true).setCompactionThreshold(topic, 400);
+        waitUntilAssert(() -> assertEquals(compactionThreshold.get(), 400));
+
+        final var readerNamespaces = ((LegacyAwareTopicPoliciesService) pulsar.getTopicPoliciesService())
+                .systemTopicService.getReaderCaches().keySet();
+        assertFalse(readerNamespaces.contains(NamespaceName.get(metaNamespace)));
+
+        // Verify Deleted events are handled
+        admin.topicPolicies(false).deleteTopicPolicies(topic);
+        waitUntilAssert(() -> assertEquals(compactionThreshold.get(), -1));
+        if (namespace.equals(metaNamespace)) {
+            assertFalse(localStore.exists(MetadataStoreTopicPoliciesService.pathFor(topicName, false)).get());
+            assertFalse(configurationStore.exists(MetadataStoreTopicPoliciesService.pathFor(topicName, true)).get());
+        }
+    }
+
+    @Test
+    public void testUserCreatedEventsTopicAreIgnored() throws Exception {
+        final var topic = TopicName.get(metaNamespace + "/" + System.currentTimeMillis()).toString();
+        admin.topics().createNonPartitionedTopic(topic);
+        admin.topicPolicies().setCompactionThreshold(topic, 1);
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies().getCompactionThreshold(topic), 1));
+
+        final var eventsTopic = metaNamespace + "/" + SystemTopicNames.NAMESPACE_EVENTS_LOCAL_NAME;
+        admin.topics().createNonPartitionedTopic(eventsTopic);
+        // Even if the __change_events topic is created, since it has detected the namespace didn't have the events
+        // topic before, it will be ignored and the policies are still read from metadata store.
+        waitUntilAssert(() -> assertEquals(admin.topicPolicies().getCompactionThreshold(topic), 1));
+        admin.topics().delete(eventsTopic);
+    }
+
+    private static void waitUntilAssert(ThrowingRunnable assertion) {
+        Awaitility.await().atMost(Duration.ofSeconds(1)).untilAsserted(assertion);
+    }
+}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
index 6b050e8b421..a0fdf039294 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
@@ -109,7 +109,7 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
         CompletableFuture<Void> f = CompletableFuture.completedFuture(null).thenRunAsync(() -> {
             for (int i = 0; i < 100; i++) {
                 TopicPolicyListener listener = new TopicPolicyListenerImpl();
-                systemTopicBasedTopicPoliciesService.registerListener(topicName, listener);
+                systemTopicBasedTopicPoliciesService.registerListenerAsync(topicName, listener);
                 Assert.assertNotNull(systemTopicBasedTopicPoliciesService.listeners.get(topicName));
                 Assert.assertTrue(systemTopicBasedTopicPoliciesService.listeners.get(topicName).size() >= 1);
                 systemTopicBasedTopicPoliciesService.unregisterListener(topicName, listener);
@@ -118,7 +118,7 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
 
         for (int i = 0; i < 100; i++) {
             TopicPolicyListener listener = new TopicPolicyListenerImpl();
-            systemTopicBasedTopicPoliciesService.registerListener(topicName, listener);
+            systemTopicBasedTopicPoliciesService.registerListenerAsync(topicName, listener);
             Assert.assertNotNull(systemTopicBasedTopicPoliciesService.listeners.get(topicName));
             Assert.assertTrue(systemTopicBasedTopicPoliciesService.listeners.get(topicName).size() >= 1);
             systemTopicBasedTopicPoliciesService.unregisterListener(topicName, listener);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyTestUtils.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyTestUtils.java
index 6b9735d59b2..7e9c697fb5d 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyTestUtils.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyTestUtils.java
@@ -72,6 +72,13 @@ public class TopicPolicyTestUtils {
     public static Optional<TopicPolicies> getTopicPoliciesBypassCache(TopicPoliciesService topicPoliciesService,
                                                                       TopicName topicName, boolean isGlobal)
             throws Exception {
+        if (topicPoliciesService instanceof LegacyAwareTopicPoliciesService legacyService) {
+            TopicPoliciesService resolved = legacyService.resolveService(topicName.getNamespaceObject()).get();
+            return getTopicPoliciesBypassCache(resolved, topicName, isGlobal);
+        }
+        if (topicPoliciesService instanceof MetadataStoreTopicPoliciesService metadataStoreService) {
+            return metadataStoreService.getTopicPoliciesDirectFromStore(topicName, isGlobal).get();
+        }
         @Cleanup final var reader = ((SystemTopicBasedTopicPoliciesService) topicPoliciesService)
                 .getNamespaceEventsSystemTopicFactory()
                 .createTopicPoliciesSystemTopicClient(topicName.getNamespaceObject())


(pulsar) 01/02: [feat][pip] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25547)

Posted by xy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

BewareMyPower pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit adc4fe4d3d15343611df464ffbf0b9ebf6e901be
Author: Yunze Xu <xy...@163.com>
AuthorDate: Mon Apr 27 14:43:00 2026 +0800

    [feat][pip] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies (#25547)
    
    (cherry picked from commit 33fe7559b4d1a2cfb9d35756d38f7bad72b54309)
---
 pip/pip-469.md | 270 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 270 insertions(+)

diff --git a/pip/pip-469.md b/pip/pip-469.md
new file mode 100644
index 00000000000..69036381f0f
--- /dev/null
+++ b/pip/pip-469.md
@@ -0,0 +1,270 @@
+# PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies
+
+# Background knowledge
+
+Apache Pulsar introduced topic-level policies in [PIP-39](pip-39.md). A broker reads and writes these policies through
+`TopicPoliciesService`. The default implementation,
+`SystemTopicBasedTopicPoliciesService`, persists topic policy changes in the namespace `__change_events` system topic and
+keeps an in-memory cache on brokers that own bundles for that namespace.
+
+[PIP-92](pip-92.md) extended topic policies with the distinction between local and global policies. Any
+`TopicPoliciesService` implementation therefore needs to handle two independent values for the same topic: the
+cluster-local policy state and the globally visible policy state.
+
+[PIP-376](pip-376.md) made `TopicPoliciesService` pluggable through the broker configuration
+`topicPoliciesServiceClassName`. That change removed the hard coupling between topic policies and system topics, but the
+backend choice is still broker-wide. During upgrade from the default system-topic backend, brokers still need a way to
+recognize namespaces that already have topic-policies state in `__change_events`, so those namespaces do not silently
+move to another backend.
+
+# Motivation
+
+The system-topic-based topic policies implementation works by appending topic policy changes to a `__change_events`
+topic in each namespace. It works well when this topic has already been loaded by a broker, then all topic policies
+operations just access the in-memory cache. However, in cold start scenarios, for example when the owner broker is down
+during a restart, the new owner broker has to create a reader on the `__change_events` topic and wait for it to catch
+up before it can read any topic policies, which is required in the path of loading a topic in the same namespace. This
+adds significant latency to the topic load path, especially before the topic is compacted.
+
+Things become worse when many `__change_events` topics move to a restarting broker. The new owner broker has to create
+many readers and replay all messages on these topics. This leads to high pressure on BookKeeper and can cause
+`Too many requests on the same bookie` errors in `GetLastMessageId` RPCs.
+
+A metadata-store-backed topic policies backend is attractive because it removes the extra lifecycle and operational
+dependency of a dedicated `__change_events` topic. A metadata-cache-based implementation can still provide caching and
+change notifications, while avoiding the cold-start latency of waiting for a system-topic reader to initialize and
+catch up.
+
+There is a second operational requirement: operators need a safe gradual rollout path. Existing namespaces that already
+have topic-policies state in `__change_events` must stay on the system-topic backend, while newly created namespaces
+should be able to use the broker-configured backend. This does not require a new namespace policy. For the upgrade case
+from the default configuration, the existence of `__change_events` is already a conservative legacy marker.
+
+# Goals
+
+## In Scope
+
+- Add a metadata-store-backed `TopicPoliciesService` implementation that does not depend on system topics.
+- Add routing logic that forces the system-topic backend for namespaces that already have `__change_events`.
+- Keep using the broker-level `topicPoliciesServiceClassName` for namespaces that do not have `__change_events`,
+  including newly created namespaces.
+
+## Out of Scope
+
+- Adding a migration framework that moves topic policies data between backends automatically.
+
+# High Level Design
+
+When topic-level policies are enabled, the broker instantiates a `LegacyAwareTopicPoliciesService` instead of using the
+configured implementation directly.
+
+The wrapper always has access to two backends:
+
+- `SystemTopicBasedTopicPoliciesService`
+- The broker-configured `topicPoliciesServiceClassName`
+
+For each namespace, the wrapper checks whether the topic-policies system topic `persistent://{tenant}/{namespace}/__change_events`
+already exists:
+
+- If it exists, the namespace is treated as a legacy system-topic namespace and all topic-policies operations are
+  routed to `SystemTopicBasedTopicPoliciesService`.
+- If it does not exist, the namespace uses the broker-configured `topicPoliciesServiceClassName`.
+
+This rule is intentionally conservative. If `__change_events` exists, the broker assumes that namespace may already
+contain topic-policies state in the system-topic backend and therefore must not be moved implicitly.
+
+This proposal also introduces `MetadataStoreTopicPoliciesService`, a concrete `TopicPoliciesService` implementation
+that stores topic policies in dedicated metadata-store paths:
+
+- Global topic policies are stored in the configuration metadata store.
+- Local topic policies are stored in the local metadata store.
+
+This keeps the storage scope aligned with the semantics introduced by PIP-92 and avoids writing topic policies through
+managed-ledger metadata side effects.
+
+# Detailed Design
+
+## Design & Implementation Details
+
+### Startup and validation
+
+`PulsarService#initTopicPoliciesService()` continues to respect `topicLevelPoliciesEnabled`. When topic-level policies
+are disabled, behavior is unchanged and `TopicPoliciesService.DISABLED` is used.
+
+When topic-level policies are enabled, the broker constructs:
+
+```java
+new LegacyAwareTopicPoliciesService(
+        this,
+        new SystemTopicBasedTopicPoliciesService(this),
+        configuredTopicPoliciesService)
+```
+
+Broker startup validates both backends:
+
+- `SystemTopicBasedTopicPoliciesService` must be instantiable.
+- The configured `topicPoliciesServiceClassName` must be instantiable.
+
+If either backend cannot be instantiated or started, broker startup fails. There is no per-request fallback from one
+backend to another.
+
+### Namespace-scoped service routing
+
+`LegacyAwareTopicPoliciesService` is responsible for:
+
+- Checking whether `__change_events` exists for the namespace by using
+  `NamespaceEventsSystemTopicFactory.checkSystemTopicExists(namespace, EventType.TOPIC_POLICY, pulsarService)`.
+- Routing `getTopicPoliciesAsync`, `updateTopicPoliciesAsync`, `deleteTopicPoliciesAsync`, and listener operations to
+  the system-topic backend when the system topic exists.
+- Routing the same operations to the configured backend when the system topic does not exist.
+
+The system-topic existence check can be cached per namespace in memory, but the routing rule is defined by actual topic
+existence rather than by new namespace metadata.
+
+This means:
+
+- Existing namespaces that already materialized `__change_events` continue to use the system-topic backend.
+- Namespaces that never created `__change_events` use the broker-configured backend.
+- Newly created namespaces use the broker-configured backend because `__change_events` does not exist yet.
+
+If `__change_events` is later deleted, the namespace falls back to the broker-configured backend on subsequent
+resolution. This matches current system-topic behavior, which already treats a missing `__change_events` topic as
+meaning the system-topic-backed topic-policies state is gone.
+
+### Metadata-backed topic policies service
+
+`MetadataStoreTopicPoliciesService` implements `TopicPoliciesService` with the following storage model:
+
+- Topic names are normalized to the partitioned topic name, so all partitions share the same topic-policies record.
+- Global policies are stored in the configuration metadata store path:
+  `/admin/topic-policies/{tenant}/{namespace}/{domain}/{encodedTopic}`.
+- Local policies are stored in the local metadata store path:
+  `/admin/local-policies/topic-policies/{tenant}/{namespace}/{domain}/{encodedTopic}`.
+
+Each node stores a serialized `TopicPolicies` document. The backend writes and reads the two scopes independently:
+
+- Reads with `GetType.GLOBAL_ONLY` only touch the global path and return a `TopicPolicies` object whose `isGlobal`
+  flag is `true`.
+- Reads with `GetType.LOCAL_ONLY` only touch the local path and return a `TopicPolicies` object whose `isGlobal` flag
+  is `false`.
+- Updates with `isGlobalPolicy=true` only modify the global path.
+- Updates with `isGlobalPolicy=false` only modify the local path.
+
+Deletes remove the local record and, unless `keepGlobalPoliciesAfterDeleting` is set, also remove the global record.
+This matches the existing `TopicPoliciesService` deletion contract.
+
+This design intentionally uses dedicated metadata nodes instead of piggybacking on `PartitionedTopicMetadata` or
+`ManagedLedgerInfo`. That keeps local/global visibility correct and avoids losing topic policies during normal
+managed-ledger metadata updates.
+
+### Listener behavior
+
+The backend registers watchers on both metadata stores:
+
+- A change on the local path re-reads the local node and notifies listeners with the latest local `TopicPolicies` or
+  `null` if the local node was removed.
+- A change on the global path re-reads the global node and notifies listeners with the latest global `TopicPolicies`
+  or `null` if the global node was removed.
+
+This preserves runtime updates for already loaded topics, including global topic policies. The backend does not add an
+append-only replay log; it relies on metadata-store notifications and read-after-notify refresh.
+
+## Public-facing Changes
+
+### Public API
+
+No new namespace policy field is introduced.
+
+No new namespace admin REST endpoint or Java admin client method is introduced.
+
+Changing the topic-policies backend for a namespace is not a public operation in this proposal. The routing rule is
+derived from `__change_events` existence plus the broker-level configuration.
+
+### Binary protocol
+
+No binary protocol changes.
+
+### Configuration
+
+- `topicPoliciesServiceClassName`
+  - Continues to define the broker-configured `TopicPoliciesService` implementation.
+  - Namespaces that do not have `__change_events` use this backend.
+  - Namespaces that already have `__change_events` keep using `SystemTopicBasedTopicPoliciesService` regardless of
+    this value.
+
+### CLI
+
+No CLI change in this proposal.
+
+### Metrics
+
+No new metric is required.
+
+# Backward & Forward Compatibility
+
+## Upgrade
+
+The intended upgrade flow is:
+
+1. Upgrade brokers to a version that understands legacy-aware backend routing.
+2. Change `topicPoliciesServiceClassName` to the alternate backend if newly created namespaces should use it.
+3. Existing namespaces that already have `__change_events` continue to use `SystemTopicBasedTopicPoliciesService`.
+4. Namespaces that do not have `__change_events`, including newly created namespaces, use the configured backend.
+
+No namespace metadata backfill is required.
+
+This upgrade rule is intentionally conservative:
+
+- If `__change_events` exists, the namespace stays on the system-topic backend.
+- If `__change_events` does not exist, the namespace uses the configured backend.
+
+This means some namespaces with an empty but already-created `__change_events` topic may continue using the
+system-topic backend. That is acceptable because it avoids missing legacy state.
+
+## Downgrade / Rollback
+
+Rolling back to a broker version that does not understand legacy-aware routing returns topic-policies backend
+selection to pure broker-wide behavior.
+
+- The older broker will no longer special-case namespaces that have `__change_events`.
+- Operators will need to choose one broker-wide backend for the rollback cluster, or migrate data before rollback if
+  both legacy system-topic namespaces and metadata-store namespaces must coexist.
+
+## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations
+
+This proposal does not introduce a new geo-replication protocol for topic policies.
+
+- Global topic policies stay in the configuration metadata store and therefore keep global visibility semantics.
+- Local topic policies stay in the local metadata store and therefore keep cluster-local visibility semantics.
+- Legacy namespaces are recognized by the existence of `__change_events`, which is already shared broker-visible topic
+  metadata.
+
+# Alternatives
+
+## Keep a single broker-wide topic policies backend
+
+This keeps the implementation simpler, but it does not solve the operational requirement to keep existing namespaces on
+their current backend while directing newly created namespaces to a different one.
+
+## Persist an explicit namespace backend marker
+
+This would also solve the upgrade problem, but it introduces new namespace-scoped metadata changes that are not
+necessary for the default-system-topic upgrade path. The proposal prefers to reuse the already existing
+`__change_events` artifact as the legacy marker.
+
+## Add a user-managed namespace override API
+
+This provides more flexibility than needed, but it also reintroduces runtime switching, rollback ambiguity, and the
+risk of one namespace being served by different backends if brokers do not resolve the override identically. The
+proposal intentionally avoids this surface.
+
+# General Notes
+
+This proposal is a follow-up to [PIP-376](pip-376.md). It keeps backend selection pluggable, but handles upgrade from
+the legacy system-topic backend by reusing `__change_events` as the compatibility marker instead of introducing a new
+namespace-level policy or namespace-level metadata field.
+
+# Links
+
+* Mailing List discussion thread: https://lists.apache.org/thread/sn2pyyl9p1vm5vr8j8qssxbbksm2bzfr
+* Mailing List voting thread: https://lists.apache.org/thread/b5mfqrmxcwwzjbkhzv6t6t12gtvjz1so