You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/06/22 12:03:30 UTC
(pulsar) branch branch-4.0 updated (131605d4e960 -> 95ce1733b623)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a change to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
from 131605d4e960 [fix][build][branch-4.0] Upgrade docker/setup-qemu-action to v4.1.0
new 9124855f8436 [fix][broker] Prevent subscribe rate limit from stalling compaction and blocking forced deletion (#26015)
new 55d299d0ba01 [fix][broker] Fix forced topic/namespace deletion hanging or failing when compaction is in progress (#26016)
new eac25231f52f [fix][broker] Fix forced topic/namespace deletion still hanging when the compaction reader reconnect stalls (#26026)
new d42ac7a2d8d0 [fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable (#26025)
new c860f44350df [fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl (#26031)
new 75f2411101bb [improve][fn] Upgrade pulsar-client-python to 3.12.0 (#26033)
new 80e00b291055 [fix][fn] Make exclusiveLeaderProducer volatile in FunctionMetaDataManager (#26046)
new b1dec33d5614 [fix][broker] Run topic policy notifications on the topic-ordered executor (#26042)
new ba47565904e6 [fix][broker] Run the message expiry check off the topic policy update path (#26040)
new c1d6ffe780d3 [fix][test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader (#26049)
new 2d7b21d417ca [fix][broker] Prevent topic policy initialization race with a buffering listener wrapper (#26044)
new a08f11b5ff9c [fix][broker] Avoid blocking metadata read on the IO thread when redirecting migrated producers/consumers (#26051)
new e192e2b7de5d [fix][proxy] Avoid blocking the proxy IO thread on a cold broker cache (#26052)
new 8ca3fd0d890c [fix][broker] Avoid blocking the dispatcher close path on delayed-delivery tracker close (#26053)
new 9f4d11e63d83 [fix][broker] Avoid blocking the bundle-throughput lookup on per-bundle metadata reads (#26054)
new a78f38a4eb94 [fix][sec] Upgrade jline to 4.2.1 and picocli to 4.7.7, drop unused jline2 (#26068)
new 95ce1733b623 [fix] functions: Run worker leader-election off the consumer event-listener thread (#26059)
The 17 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
conf/broker.conf | 7 +
conf/standalone.conf | 7 +
distribution/server/pom.xml | 6 -
distribution/server/src/assemble/LICENSE.bin.txt | 7 +-
distribution/shell/src/assemble/LICENSE.bin.txt | 6 +-
docker/pulsar/Dockerfile | 2 +-
pom.xml | 7 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 13 ++
.../broker/resources/MetadataStoreCacheLoader.java | 45 ++--
.../resources/MetadataStoreCacheLoaderTest.java | 76 +++++++
.../broker/delayed/DelayedDeliveryTracker.java | 10 +
.../bucket/BucketDelayedDeliveryTracker.java | 27 ++-
.../pulsar/broker/loadbalance/LoadManager.java | 52 +++--
.../broker/loadbalance/ModularLoadManager.java | 14 ++
.../loadbalance/impl/ModularLoadManagerImpl.java | 89 ++++----
.../broker/service/AbstractBaseDispatcher.java | 3 +-
.../pulsar/broker/service/AbstractTopic.java | 33 +--
.../pulsar/broker/service/BrokerService.java | 46 ++++-
.../org/apache/pulsar/broker/service/Consumer.java | 22 +-
.../apache/pulsar/broker/service/ServerCnx.java | 164 +++++++++------
.../SystemTopicBasedTopicPoliciesService.java | 228 +++++++++++++++++----
.../broker/service/TopicPolicyListenerWrapper.java | 117 +++++++++++
.../service/nonpersistent/NonPersistentTopic.java | 50 +++--
.../PersistentDispatcherMultipleConsumers.java | 8 +-
...rsistentDispatcherMultipleConsumersClassic.java | 8 +-
.../broker/service/persistent/PersistentTopic.java | 199 ++++++++++++------
.../apache/pulsar/client/impl/RawReaderImpl.java | 31 +++
.../common/naming/NamespaceBundleFactory.java | 31 +--
.../impl/ModularLoadManagerImplTest.java | 2 +-
.../broker/namespace/NamespaceServiceTest.java | 2 +-
.../pulsar/broker/service/MessageTTLTest.java | 7 +-
.../SystemTopicBasedTopicPoliciesServiceTest.java | 180 ++++++++++++++--
.../service/TopicPolicyListenerWrapperTest.java | 124 +++++++++++
.../apache/pulsar/client/impl/RawReaderTest.java | 82 ++++++++
.../apache/pulsar/compaction/CompactionTest.java | 124 ++++++++++-
.../apache/pulsar/client/impl/ConsumerImpl.java | 4 +
.../functions/worker/FunctionMetaDataManager.java | 2 +-
.../pulsar/functions/worker/LeaderService.java | 28 ++-
.../pulsar/functions/worker/LeaderServiceTest.java | 8 +
39 files changed, 1500 insertions(+), 371 deletions(-)
create mode 100644 pulsar-broker-common/src/test/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoaderTest.java
create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapper.java
create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapperTest.java
(pulsar) 10/17: [fix][test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader (#26049)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit c1d6ffe780d380954e811d077852a823c5455952
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Thu Jun 18 02:44:55 2026 +0300
[fix][test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader (#26049)
(cherry picked from commit 6f82106ea6c4d508a883ff27875ef279c7c4101c)
---
.../SystemTopicBasedTopicPoliciesServiceTest.java | 29 ++++++++++++++++------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
index dbce03b9d1bb..04b9bc67e7e0 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
@@ -51,6 +51,7 @@ import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
import org.apache.pulsar.broker.systopic.SystemTopicClient;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.events.PulsarEvent;
import org.apache.pulsar.common.naming.NamespaceName;
@@ -559,14 +560,28 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
assertTrue(logFound);
});
-
- // Since cleanPoliciesCacheInitMap() is executed, should add the failed reader into readerCache again.
- // Then in SystemTopicBasedTopicPoliciesService, readerCache has a closed reader,
- // and policyCacheInitMap do not contain a future.
- // To simulate the situation: when getTopicPolicy() execute, it will do prepareInitPoliciesCacheAsync() and
- // use a closed reader to read the __change_event topic. Then throw exception
- spyReaderCaches.put(NamespaceName.get(NAMESPACE5), readerCompletableFuture);
+ // The reader.close() above drives readMorePoliciesAsync() into cleanPoliciesCacheInitMap(), which logs
+ // "Closing the topic policies reader for" and removes the namespace from policyCacheInitMap and readerCaches.
+ // Now exercise a follow-up prepareInitPoliciesCacheAsync() that reuses a closed reader and must fail in
+ // initPolicesCache(). Two timing hazards made this flaky (#25081), so both are pinned deterministically:
+ // 1) A real closed reader's hasMoreEventsAsync() (Reader.hasMessageAvailableAsync()) can answer from cached
+ // state instead of failing with AlreadyClosedException, in which case initPolicesCache() reaches the end
+ // of the topic and completes successfully and the expected exception never happens. Use a spy whose
+ // hasMoreEventsAsync() always fails with AlreadyClosedException.
+ // 2) A background topic load can re-run prepareInitPoliciesCacheAsync() for the namespace and leave a
+ // completed init future behind; the next call would then short-circuit through the existing-future
+ // branch and never re-initialize. Drop any init future right before the call so it re-initializes, and
+ // stub createSystemTopicClient() so every reader created for this namespace is the failing spy — then
+ // whichever initialization wins the race fails in the same (asserted) way.
+ SystemTopicClient.Reader<PulsarEvent> closedReader = Mockito.spy(reader);
+ Mockito.doReturn(CompletableFuture.failedFuture(
+ new PulsarClientException.AlreadyClosedException("Reader is already closed")))
+ .when(closedReader).hasMoreEventsAsync();
+ Mockito.doReturn(CompletableFuture.completedFuture(closedReader))
+ .when(spyService).createSystemTopicClient(NamespaceName.get(NAMESPACE5));
+ spyReaderCaches.put(NamespaceName.get(NAMESPACE5), CompletableFuture.completedFuture(closedReader));
FieldUtils.writeDeclaredField(spyService, "readerCaches", spyReaderCaches, true);
+ spyService.policyCacheInitMap.remove(NamespaceName.get(NAMESPACE5));
CompletableFuture<Boolean> prepareFuture = new CompletableFuture<>();
try {
(pulsar) 13/17: [fix][proxy] Avoid blocking the proxy IO thread on a cold broker cache (#26052)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e192e2b7de5d2c6f6da8ffda966be616ccdf2486
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Thu Jun 18 04:20:44 2026 -0700
[fix][proxy] Avoid blocking the proxy IO thread on a cold broker cache (#26052)
(cherry picked from commit 1d7ae01479cc225b9f588c6727d6c5d2e713a0e3)
---
.../broker/resources/MetadataStoreCacheLoader.java | 45 +++++++------
.../resources/MetadataStoreCacheLoaderTest.java | 76 ++++++++++++++++++++++
2 files changed, 98 insertions(+), 23 deletions(-)
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoader.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoader.java
index 29451148da44..7a0a41282824 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoader.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoader.java
@@ -25,7 +25,7 @@ import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
-import java.util.function.Supplier;
+import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.bookkeeper.common.util.OrderedScheduler;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.pulsar.common.util.FutureUtil;
@@ -46,6 +46,7 @@ public class MetadataStoreCacheLoader implements Closeable {
private volatile List<LoadManagerReport> availableBrokers;
private final FutureUtil.Sequencer<Void> sequencer;
+ private final AtomicBoolean refreshInProgress = new AtomicBoolean(false);
private final OrderedScheduler orderedExecutor = OrderedScheduler.newSchedulerBuilder().numThreads(8)
.name("pulsar-metadata-cache-loader-ordered-cache").build();
@@ -65,40 +66,38 @@ public class MetadataStoreCacheLoader implements Closeable {
* @throws Exception
*/
public void init() throws Exception {
- Supplier<CompletableFuture<Void>> tryUpdate = () -> {
- return loadReportResources.getChildrenAsync(LOADBALANCE_BROKERS_ROOT)
- .thenComposeAsync(brokerNodes -> {
- return updateBrokerList(brokerNodes).thenRun(() -> {
- log.info("Successfully updated broker info {}", brokerNodes);
- });
- })
- .exceptionally(ex -> {
- log.warn("Error updating broker info after broker list changed", ex);
- return null;
- });
- };
loadReportResources.getStore().registerListener((n) -> {
if (LOADBALANCE_BROKERS_ROOT.equals(n.getPath()) && NotificationType.ChildrenChanged.equals(n.getType())) {
- sequencer.sequential(tryUpdate);
+ sequencer.sequential(this::reloadBrokers);
}
});
if (loadReportResources.getStore() instanceof MetadataStoreExtended) {
((MetadataStoreExtended) loadReportResources.getStore()).registerSessionListener(sessionEvent ->
- sequencer.sequential(tryUpdate));
+ sequencer.sequential(this::reloadBrokers));
}
// Do initial fetch of brokers list
- tryUpdate.get().get(operationTimeoutMs, TimeUnit.MILLISECONDS);
+ reloadBrokers().get(operationTimeoutMs, TimeUnit.MILLISECONDS);
+ }
+
+ private CompletableFuture<Void> reloadBrokers() {
+ return loadReportResources.getChildrenAsync(LOADBALANCE_BROKERS_ROOT)
+ .thenComposeAsync(brokerNodes -> updateBrokerList(brokerNodes).thenRun(() ->
+ log.info("Successfully updated broker info {}", brokerNodes)))
+ .exceptionally(ex -> {
+ log.warn("Error updating broker info after broker list changed", ex);
+ return null;
+ });
}
public List<LoadManagerReport> getAvailableBrokers() {
- if (CollectionUtils.isEmpty(availableBrokers)) {
- try {
- updateBrokerList(loadReportResources.getChildren(LOADBALANCE_BROKERS_ROOT));
- } catch (Exception e) {
- log.warn("Error updating broker from zookeeper.", e);
- }
+ List<LoadManagerReport> brokers = availableBrokers;
+ if (CollectionUtils.isEmpty(brokers) && refreshInProgress.compareAndSet(false, true)) {
+ // Avoid blocking the caller (which may be a Netty IO thread): refresh the cache in the
+ // background and return the current snapshot. The cache is otherwise kept up to date by the
+ // metadata-store listener, so an empty snapshot means there are no active brokers.
+ sequencer.sequential(this::reloadBrokers).whenComplete((__, ex) -> refreshInProgress.set(false));
}
- return availableBrokers;
+ return brokers == null ? new ArrayList<>() : brokers;
}
@Override
diff --git a/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoaderTest.java b/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoaderTest.java
new file mode 100644
index 000000000000..e8d4d28153c9
--- /dev/null
+++ b/pulsar-broker-common/src/test/java/org/apache/pulsar/broker/resources/MetadataStoreCacheLoaderTest.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.resources;
+
+import static org.apache.pulsar.broker.resources.MetadataStoreCacheLoader.LOADBALANCE_BROKERS_ROOT;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import lombok.Cleanup;
+import org.apache.pulsar.metadata.api.MetadataStore;
+import org.apache.pulsar.policies.data.loadbalancer.LoadManagerReport;
+import org.testng.annotations.Test;
+
+public class MetadataStoreCacheLoaderTest {
+
+ private MetadataStoreCacheLoader newCacheLoader(LoadManagerReportResources loadReportResources) throws Exception {
+ MetadataStore store = mock(MetadataStore.class);
+ PulsarResources pulsarResources = mock(PulsarResources.class);
+ when(pulsarResources.getLoadReportResources()).thenReturn(loadReportResources);
+ when(loadReportResources.getStore()).thenReturn(store);
+ return new MetadataStoreCacheLoader(pulsarResources, 5000);
+ }
+
+ @Test
+ public void testGetAvailableBrokersServesCacheWithoutBlocking() throws Exception {
+ LoadManagerReportResources loadReportResources = mock(LoadManagerReportResources.class);
+ LoadManagerReport report = mock(LoadManagerReport.class);
+ when(loadReportResources.getChildrenAsync(LOADBALANCE_BROKERS_ROOT))
+ .thenReturn(CompletableFuture.completedFuture(List.of("broker-1")));
+ when(loadReportResources.getAsync(LOADBALANCE_BROKERS_ROOT + "/broker-1"))
+ .thenReturn(CompletableFuture.completedFuture(Optional.of(report)));
+
+ @Cleanup
+ MetadataStoreCacheLoader loader = newCacheLoader(loadReportResources);
+
+ assertEquals(loader.getAvailableBrokers(), List.of(report));
+ // The blocking, synchronous getChildren(...) must never be used: it could stall a Netty IO thread.
+ verify(loadReportResources, never()).getChildren(anyString());
+ }
+
+ @Test
+ public void testGetAvailableBrokersDoesNotBlockOnEmptyCache() throws Exception {
+ LoadManagerReportResources loadReportResources = mock(LoadManagerReportResources.class);
+ when(loadReportResources.getChildrenAsync(LOADBALANCE_BROKERS_ROOT))
+ .thenReturn(CompletableFuture.completedFuture(List.of()));
+
+ @Cleanup
+ MetadataStoreCacheLoader loader = newCacheLoader(loadReportResources);
+
+ assertTrue(loader.getAvailableBrokers().isEmpty());
+ verify(loadReportResources, never()).getChildren(anyString());
+ }
+}
(pulsar) 16/17: [fix][sec] Upgrade jline to 4.2.1 and picocli to 4.7.7, drop unused jline2 (#26068)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a78f38a4eb949508fea50934848d46485678e6c7
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sat Jun 20 13:51:53 2026 +0300
[fix][sec] Upgrade jline to 4.2.1 and picocli to 4.7.7, drop unused jline2 (#26068)
(cherry picked from commit c536dcbd4ce598123dbbb967d1d37788c9a0b8a8)
---
distribution/server/pom.xml | 6 ------
distribution/server/src/assemble/LICENSE.bin.txt | 7 +++----
distribution/shell/src/assemble/LICENSE.bin.txt | 6 +++---
pom.xml | 5 ++---
4 files changed, 8 insertions(+), 16 deletions(-)
diff --git a/distribution/server/pom.xml b/distribution/server/pom.xml
index 94c858a6f461..8255ab972e72 100644
--- a/distribution/server/pom.xml
+++ b/distribution/server/pom.xml
@@ -93,12 +93,6 @@
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>jline</groupId>
- <artifactId>jline</artifactId>
- <version>${jline.version}</version>
- </dependency>
-
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-package-bookkeeper-storage</artifactId>
diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index da1ddf3704c1..1c5b15d1a548 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -245,8 +245,8 @@ This projects includes binary packages with the following licenses:
The Apache Software License, Version 2.0
* JCommander -- com.beust-jcommander-1.82.jar
* Picocli
- - info.picocli-picocli-4.7.5.jar
- - info.picocli-picocli-shell-jline3-4.7.5.jar
+ - info.picocli-picocli-4.7.7.jar
+ - info.picocli-picocli-shell-jline3-4.7.7.jar
* High Performance Primitive Collections for Java -- com.carrotsearch-hppc-0.9.1.jar
* Jackson
- com.fasterxml.jackson.core-jackson-annotations-2.18.6.jar
@@ -565,8 +565,7 @@ BSD 3-clause "New" or "Revised" License
- com.google.auth-google-auth-library-oauth2-http-1.24.1.jar -- ../licenses/LICENSE-google-auth-library.txt
* LevelDB -- (included in org.rocksdb.*.jar) -- ../licenses/LICENSE-LevelDB.txt
* JSR305 -- com.google.code.findbugs-jsr305-3.0.2.jar -- ../licenses/LICENSE-JSR305.txt
- * JLine -- jline-jline-2.14.6.jar -- ../licenses/LICENSE-JLine.txt
- * JLine3 -- org.jline-jline-3.21.0.jar -- ../licenses/LICENSE-JLine.txt
+ * JLine3 -- org.jline-jline-4.2.1.jar -- ../licenses/LICENSE-JLine.txt
* OW2 ASM
- org.ow2.asm-asm-9.10.jar -- ../licenses/LICENSE-ASM.txt
- org.ow2.asm-asm-commons-9.10.jar -- ../licenses/LICENSE-ASM.txt
diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt
index 2bae377ffc92..82a0828770b2 100644
--- a/distribution/shell/src/assemble/LICENSE.bin.txt
+++ b/distribution/shell/src/assemble/LICENSE.bin.txt
@@ -310,8 +310,8 @@ This projects includes binary packages with the following licenses:
The Apache Software License, Version 2.0
* Picocli
- - picocli-4.7.5.jar
- - picocli-shell-jline3-4.7.5.jar
+ - picocli-4.7.7.jar
+ - picocli-shell-jline3-4.7.7.jar
* Jackson
- jackson-annotations-2.18.6.jar
- jackson-core-2.18.6.jar
@@ -426,8 +426,8 @@ The Apache Software License, Version 2.0
* JSpecify -- jspecify-1.0.0.jar
BSD 3-clause "New" or "Revised" License
+ * JLine3 -- jline-4.2.1.jar -- ../licenses/LICENSE-JLine.txt
* JSR305 -- jsr305-3.0.2.jar -- ../licenses/LICENSE-JSR305.txt
- * JLine3 -- jline-3.21.0.jar -- ../licenses/LICENSE-JLine.txt
MIT License
* SLF4J -- ../licenses/LICENSE-SLF4J.txt
diff --git a/pom.xml b/pom.xml
index 60d188e476ba..ed357605cce0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -272,8 +272,7 @@ flexible messaging model and an intuitive client API.</description>
<javax.servlet-api>3.1.0</javax.servlet-api>
<caffeine.version>2.9.1</caffeine.version>
<java-semver.version>0.9.0</java-semver.version>
- <jline.version>2.14.6</jline.version>
- <jline3.version>3.21.0</jline3.version>
+ <jline3.version>4.2.1</jline3.version>
<hppc.version>0.9.1</hppc.version>
<spark-streaming_2.10.version>2.1.0</spark-streaming_2.10.version>
<assertj-core.version>3.27.7</assertj-core.version>
@@ -311,7 +310,7 @@ flexible messaging model and an intuitive client API.</description>
<opentelemetry.instrumentation.version>2.21.0</opentelemetry.instrumentation.version>
<opentelemetry.instrumentation.alpha.version>${opentelemetry.instrumentation.version}-alpha</opentelemetry.instrumentation.alpha.version>
<opentelemetry.semconv.version>1.37.0</opentelemetry.semconv.version>
- <picocli.version>4.7.5</picocli.version>
+ <picocli.version>4.7.7</picocli.version>
<re2j.version>1.8</re2j.version>
<completable-futures.version>0.3.6</completable-futures.version>
<failsafe.version>3.3.2</failsafe.version>
(pulsar) 11/17: [fix][broker] Prevent topic policy initialization race with a buffering listener wrapper (#26044)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 2d7b21d417ca0d161b8a1aa27d84e88a44ca8997
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Thu Jun 18 10:29:17 2026 +0300
[fix][broker] Prevent topic policy initialization race with a buffering listener wrapper (#26044)
(cherry picked from commit b70b3a386c29374cdfa9c5367aacda8a730e4a71)
---
.../pulsar/broker/service/AbstractTopic.java | 8 +-
.../broker/service/TopicPolicyListenerWrapper.java | 117 +++++++++++++++++++
.../service/nonpersistent/NonPersistentTopic.java | 18 ++-
.../broker/service/persistent/PersistentTopic.java | 56 +++++++---
.../service/TopicPolicyListenerWrapperTest.java | 124 +++++++++++++++++++++
5 files changed, 304 insertions(+), 19 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 2a950c265918..5dad5b1b0799 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -548,14 +548,18 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
&& maxProducers <= USER_CREATED_PRODUCER_COUNTER_UPDATER.get(this);
}
+ protected TopicPolicyListener getTopicPolicyListener() {
+ return this;
+ }
+
protected void registerTopicPolicyListener() {
brokerService.getPulsar().getTopicPoliciesService()
- .registerListenerAsync(TopicName.getPartitionedTopicName(topic), this);
+ .registerListenerAsync(TopicName.getPartitionedTopicName(topic), getTopicPolicyListener());
}
protected void unregisterTopicPolicyListener() {
brokerService.getPulsar().getTopicPoliciesService()
- .unregisterListener(TopicName.getPartitionedTopicName(topic), this);
+ .unregisterListener(TopicName.getPartitionedTopicName(topic), getTopicPolicyListener());
}
protected boolean isSameAddressProducersExceeded(Producer producer) {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapper.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapper.java
new file mode 100644
index 000000000000..c04d5d30e12d
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapper.java
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.broker.service;
+
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.common.policies.data.TopicPolicies;
+
+/**
+ * This TopicPolicyListener is used as a wrapper for the real TopicPolicyListener.
+ * This prevents a race condition in initialization where the topic policy state can change while the
+ * topic policy state is being applied to the topic in PersistentTopic#initTopicPolicy() method or in
+ * NonPersistentTopic#initialize method. The impact of the race conditions is that the topic policy state would
+ * be left in an inconsistent state until another update arrives. This is a rare corner case, but possible.
+ */
+@Slf4j
+public class TopicPolicyListenerWrapper implements TopicPolicyListener {
+ private final TopicPolicyListener realTopicListener;
+ // The latest value received during initialization, per scope. A null reference means no update was
+ // received during initialization (the loaded value should be used); an Optional that is present holds the
+ // received policies, and an empty Optional records that a delete (onUpdate(null)) was received, so the
+ // loaded value must not be applied. Optional is used because the map-like field cannot itself hold null
+ // while still distinguishing "not received" (null) from "received a delete" (Optional.empty()).
+ private Optional<TopicPolicies> latestGlobalPolicies;
+ private Optional<TopicPolicies> latestLocalPolicies;
+ private boolean initialized;
+ private final long createdTimestampNanos = System.nanoTime();
+ private static final long INITIALIZATION_WARNING_LOG_INTERVAL_NANOS = TimeUnit.SECONDS.toNanos(30);
+ private int lastIntervalLogged;
+
+ public TopicPolicyListenerWrapper(TopicPolicyListener realTopicListener) {
+ this.realTopicListener = realTopicListener;
+ }
+
+ @Override
+ public synchronized void onUpdate(TopicPolicies data) {
+ if (initialized) {
+ realTopicListener.onUpdate(data);
+ return;
+ }
+
+ maybeLogWarning();
+
+ // Record the latest value received during initialization so it can be applied (preferring it over the
+ // loaded value) in completeInitialization. A received value is stored as Optional.of(data) and a delete
+ // as Optional.empty(), so the delete is propagated downstream instead of being lost.
+ if (data == null) {
+ // A delete (onUpdate(null)) does not carry the global/local scope through the listener interface,
+ // so record it for both scopes; a later scoped update received during initialization still
+ // overrides its own scope.
+ latestGlobalPolicies = Optional.empty();
+ latestLocalPolicies = Optional.empty();
+ } else if (data.isGlobalPolicies()) {
+ latestGlobalPolicies = Optional.of(data);
+ } else {
+ latestLocalPolicies = Optional.of(data);
+ }
+ }
+
+ /**
+ * Complete initialization of the TopicPolicyListenerWrapper and emit the latest policies to the real listener.
+ * @param loadedGlobalPolicies the loaded global policies
+ * @param loadedLocalPolicies the loaded local policies
+ */
+ public synchronized void completeInitialization(TopicPolicies loadedGlobalPolicies,
+ TopicPolicies loadedLocalPolicies) {
+ // The listener might have received a newer value (or a delete) than the loaded one while the loading
+ // was happening; prefer the latest value received during initialization, falling back to the loaded
+ // value only when nothing was received for that scope.
+ emitInitialPolicies(latestGlobalPolicies, loadedGlobalPolicies);
+ emitInitialPolicies(latestLocalPolicies, loadedLocalPolicies);
+ latestGlobalPolicies = null;
+ latestLocalPolicies = null;
+ initialized = true;
+ }
+
+ private void emitInitialPolicies(Optional<TopicPolicies> latestReceived, TopicPolicies loaded) {
+ if (latestReceived != null) {
+ // A value (or a delete) was received during initialization; it supersedes the loaded value.
+ realTopicListener.onUpdate(latestReceived.orElse(null));
+ } else if (loaded != null) {
+ realTopicListener.onUpdate(loaded);
+ }
+ }
+
+ // warn if the initialization takes too long and updates have been received
+ // this helps detect issues where completeInitialization didn't get called after loading policies
+ private void maybeLogWarning() {
+ long durationNanos = System.nanoTime() - createdTimestampNanos;
+ int warningLogIntervalCount = (int) (durationNanos / INITIALIZATION_WARNING_LOG_INTERVAL_NANOS);
+ if (warningLogIntervalCount > lastIntervalLogged) {
+ log.warn("TopicPolicyUpdate buffered. TopicPolicyListenerWrapper initialization phase took too long. "
+ + "completeInitialization should have been called to complete the phase. "
+ + "topicPolicyListener={} sinceCreationMs={}",
+ realTopicListener, TimeUnit.NANOSECONDS.toMillis(durationNanos));
+ lastIntervalLogged = warningLogIntervalCount;
+ }
+ }
+}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index 4e04fd268155..0e61bc059ab5 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -66,6 +66,7 @@ import org.apache.pulsar.broker.service.SubscriptionOption;
import org.apache.pulsar.broker.service.Topic;
import org.apache.pulsar.broker.service.TopicAttributes;
import org.apache.pulsar.broker.service.TopicPolicyListener;
+import org.apache.pulsar.broker.service.TopicPolicyListenerWrapper;
import org.apache.pulsar.broker.service.TransportCnx;
import org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException;
import org.apache.pulsar.broker.service.schema.exceptions.NotExistSchemaException;
@@ -127,6 +128,9 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
TOPIC_ATTRIBUTES_FIELD_UPDATER = AtomicReferenceFieldUpdater.newUpdater(
NonPersistentTopic.class, TopicAttributes.class, "topicAttributes");
+ // prevents race conditions in topic policy initialization
+ private final TopicPolicyListenerWrapper topicPolicyListener = new TopicPolicyListenerWrapper(this);
+
private static class TopicStats {
public double averageMsgSize;
public double aggMsgRateIn;
@@ -179,7 +183,14 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
updatePublishRateLimiter();
updateResourceGroupLimiter(policies);
return updateClusterMigrated();
- }, getPoliciesNotifyThread());
+ }, getPoliciesNotifyThread())
+ // Complete the topic-policy listener wrapper so buffered and future topic-level policy
+ // updates are forwarded to this topic. Without this the wrapper stays uninitialized forever
+ // and all topic-level policy updates are silently dropped. Unlike PersistentTopic,
+ // non-persistent topics don't load initial topic policies (matching the previous behavior),
+ // so the loaded values are passed as null.
+ .thenRunAsync(() -> topicPolicyListener.completeInitialization(null, null),
+ getPoliciesNotifyThread());
}
@Override
@@ -1297,4 +1308,9 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
return TOPIC_ATTRIBUTES_FIELD_UPDATER.updateAndGet(this,
old -> old != null ? old : new TopicAttributes(TopicName.get(topic)));
}
+
+ @Override
+ public TopicPolicyListener getTopicPolicyListener() {
+ return topicPolicyListener;
+ }
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index dc379fa9d96c..addcfff18a2d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -60,6 +60,7 @@ import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
import java.util.function.BiConsumer;
import java.util.function.BiFunction;
+import java.util.function.Function;
import lombok.Getter;
import lombok.Value;
import org.apache.bookkeeper.client.BKException.BKNoSuchLedgerExistsException;
@@ -135,6 +136,8 @@ import org.apache.pulsar.broker.service.Subscription;
import org.apache.pulsar.broker.service.SubscriptionOption;
import org.apache.pulsar.broker.service.Topic;
import org.apache.pulsar.broker.service.TopicPoliciesService;
+import org.apache.pulsar.broker.service.TopicPolicyListener;
+import org.apache.pulsar.broker.service.TopicPolicyListenerWrapper;
import org.apache.pulsar.broker.service.TransportCnx;
import org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage;
import org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException;
@@ -292,6 +295,10 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
// Record the last time max read position is moved forward, unless it's a marker message.
@Getter
private volatile long lastMaxReadPositionMovedForwardTimestamp = 0;
+
+ // prevents race conditions in topic policy initialization
+ private final TopicPolicyListenerWrapper topicPolicyListener = new TopicPolicyListenerWrapper(this);
+
@Getter
private final ExecutorService orderedExecutor;
@@ -4756,22 +4763,34 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
final var topicPoliciesService = brokerService.pulsar().getTopicPoliciesService();
final var partitionedTopicName = TopicName.getPartitionedTopicName(topic);
- return topicPoliciesService.registerListenerAsync(partitionedTopicName, this).thenCompose(registered -> {
- if (!registered) {
- return CompletableFuture.completedFuture(null);
- }
- if (ExtensibleLoadManagerImpl.isInternalTopic(topic)) {
- return CompletableFuture.completedFuture(null);
- }
- return topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
- TopicPoliciesService.GetType.GLOBAL_ONLY)
- .thenAcceptAsync(optionalPolicies -> optionalPolicies.ifPresent(this::onUpdate),
- getPoliciesNotifyThread())
- .thenCompose(__ -> topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
- TopicPoliciesService.GetType.LOCAL_ONLY))
- .thenAcceptAsync(optionalPolicies -> optionalPolicies.ifPresent(this::onUpdate),
- getPoliciesNotifyThread());
- });
+ return topicPoliciesService.registerListenerAsync(partitionedTopicName, topicPolicyListener)
+ .thenCompose(registered -> {
+ if (!registered) {
+ return CompletableFuture.completedFuture(null);
+ }
+ if (ExtensibleLoadManagerImpl.isInternalTopic(topic)) {
+ // Internal topics don't load topic-level policies, but the listener wrapper must
+ // still be initialized so any buffered/future updates are forwarded to the topic
+ // instead of being silently dropped.
+ return CompletableFuture.runAsync(
+ () -> topicPolicyListener.completeInitialization(null, null),
+ getPoliciesNotifyThread());
+ }
+ // future for fetching global topic policies
+ CompletableFuture<Optional<TopicPolicies>> globalPoliciesFuture =
+ topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
+ TopicPoliciesService.GetType.GLOBAL_ONLY);
+ // future for fetching local topic policies
+ CompletableFuture<Optional<TopicPolicies>> localPoliciesFuture =
+ topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
+ TopicPoliciesService.GetType.LOCAL_ONLY);
+ return globalPoliciesFuture.thenCombine(localPoliciesFuture, (global, local) -> {
+ // finally update the topic policies with the latest value or loaded value
+ return CompletableFuture.runAsync(() ->
+ topicPolicyListener.completeInitialization(global.orElse(null), local.orElse(null)),
+ getPoliciesNotifyThread());
+ }).thenCompose(Function.identity());
+ });
}
@VisibleForTesting
@@ -4941,4 +4960,9 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return future;
}
+
+ @Override
+ public TopicPolicyListener getTopicPolicyListener() {
+ return topicPolicyListener;
+ }
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapperTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapperTest.java
new file mode 100644
index 000000000000..0e7553592741
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyListenerWrapperTest.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.pulsar.common.policies.data.TopicPolicies;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class TopicPolicyListenerWrapperTest {
+
+ private static TopicPolicies globalPolicies() {
+ return TopicPolicies.builder().isGlobal(true).build();
+ }
+
+ private static TopicPolicies localPolicies() {
+ return TopicPolicies.builder().isGlobal(false).build();
+ }
+
+ private static final class RecordingListener implements TopicPolicyListener {
+ final List<TopicPolicies> updates = new ArrayList<>();
+
+ @Override
+ public void onUpdate(TopicPolicies data) {
+ updates.add(data);
+ }
+ }
+
+ @Test
+ public void shouldBufferUpdatesUntilInitializedThenForwardLive() {
+ RecordingListener real = new RecordingListener();
+ TopicPolicyListenerWrapper wrapper = new TopicPolicyListenerWrapper(real);
+
+ // Updates received before initialization are buffered, not forwarded.
+ TopicPolicies bufferedLocal = localPolicies();
+ wrapper.onUpdate(bufferedLocal);
+ assertThat(real.updates).isEmpty();
+
+ // On completion, the buffered local value wins over the loaded local value; the loaded global value
+ // is applied since none was buffered.
+ TopicPolicies loadedGlobal = globalPolicies();
+ wrapper.completeInitialization(loadedGlobal, localPolicies());
+ assertThat(real.updates).containsExactly(loadedGlobal, bufferedLocal);
+
+ // After initialization, updates are forwarded immediately.
+ TopicPolicies liveUpdate = localPolicies();
+ wrapper.onUpdate(liveUpdate);
+ assertThat(real.updates).containsExactly(loadedGlobal, bufferedLocal, liveUpdate);
+ }
+
+ @Test
+ public void shouldPreferBufferedOverLoadedForBothScopes() {
+ RecordingListener real = new RecordingListener();
+ TopicPolicyListenerWrapper wrapper = new TopicPolicyListenerWrapper(real);
+
+ TopicPolicies bufferedGlobal = globalPolicies();
+ TopicPolicies bufferedLocal = localPolicies();
+ wrapper.onUpdate(bufferedGlobal);
+ wrapper.onUpdate(bufferedLocal);
+
+ wrapper.completeInitialization(globalPolicies(), localPolicies());
+ assertThat(real.updates).containsExactly(bufferedGlobal, bufferedLocal);
+ }
+
+ @Test
+ public void shouldApplyLoadedWhenNothingBuffered() {
+ RecordingListener real = new RecordingListener();
+ TopicPolicyListenerWrapper wrapper = new TopicPolicyListenerWrapper(real);
+
+ TopicPolicies loadedGlobal = globalPolicies();
+ TopicPolicies loadedLocal = localPolicies();
+ wrapper.completeInitialization(loadedGlobal, loadedLocal);
+ assertThat(real.updates).containsExactly(loadedGlobal, loadedLocal);
+ }
+
+ @Test
+ public void shouldSuppressLoadedValuesWhenDeletedBeforeInitialization() {
+ RecordingListener real = new RecordingListener();
+ TopicPolicyListenerWrapper wrapper = new TopicPolicyListenerWrapper(real);
+
+ // A delete (null) arriving before initialization must not NPE and must not be forwarded yet (#26037).
+ assertThatCode(() -> wrapper.onUpdate(null)).doesNotThrowAnyException();
+ assertThat(real.updates).isEmpty();
+
+ // The delete supersedes the (now-stale) loaded values: they are not applied, and the delete (null) is
+ // propagated downstream instead.
+ wrapper.completeInitialization(globalPolicies(), localPolicies());
+ assertThat(real.updates).containsExactly(null, null);
+ }
+
+ @Test
+ public void shouldApplyLatestScopedUpdateOverEarlierDeleteDuringInitialization() {
+ RecordingListener real = new RecordingListener();
+ TopicPolicyListenerWrapper wrapper = new TopicPolicyListenerWrapper(real);
+
+ // A delete records empty for both scopes, then a newer global update overrides only the global scope.
+ wrapper.onUpdate(null);
+ TopicPolicies newerGlobal = globalPolicies();
+ wrapper.onUpdate(newerGlobal);
+
+ wrapper.completeInitialization(globalPolicies(), localPolicies());
+ // Global: the newer update wins; Local: the delete (null) wins over the loaded local value.
+ assertThat(real.updates).containsExactly(newerGlobal, null);
+ }
+}
(pulsar) 03/17: [fix][broker] Fix forced topic/namespace deletion still hanging when the compaction reader reconnect stalls (#26026)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit eac25231f52f3a5dc6795235245707caac14071f
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Jun 14 14:25:57 2026 +0300
[fix][broker] Fix forced topic/namespace deletion still hanging when the compaction reader reconnect stalls (#26026)
(cherry picked from commit 2325d1eb17d090afc0bd5eaeaa5c2a2e6a4445cf)
---
.../broker/service/persistent/PersistentTopic.java | 20 ++++---
.../apache/pulsar/client/impl/RawReaderImpl.java | 20 +++++++
.../apache/pulsar/client/impl/RawReaderTest.java | 63 ++++++++++++++++++++++
.../apache/pulsar/compaction/CompactionTest.java | 37 +++++++++++++
.../apache/pulsar/client/impl/ConsumerImpl.java | 4 ++
5 files changed, 138 insertions(+), 6 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 38884047f2b9..d68e21740502 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -1370,12 +1370,20 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return;
}
}
- // Unsubscribe compaction cursor and delete compacted ledger. Wait for any in-flight compaction to finish
- // first, but don't let a compaction that completed exceptionally block the cursor deletion: the deletion
- // would otherwise fail on every retry until the topic instance is reloaded (issue #24148). Note that a
- // fenced topic makes the compactor's reader fail with an unrecoverable error, so a forced deletion
- // terminates an in-flight compaction exceptionally rather than waiting for it to complete normally.
- currentCompaction.exceptionally(compactionEx -> {
+ // Unsubscribe compaction cursor and delete compacted ledger. Normally we wait for any in-flight compaction
+ // to finish first, but a compaction that completed exceptionally must not block the cursor deletion: it
+ // would otherwise fail on every retry until the topic instance is reloaded (issue #24148).
+ //
+ // Moreover, when the topic is being closed or deleted it is already fenced and any in-flight compaction is
+ // being aborted. Waiting for that compaction to complete is both unnecessary and unsafe here: the
+ // compactor's reader is expected to fail once the topic is fenced, but that depends on how the client
+ // reconnect surfaces the failure (a retriable lookup-stage error keeps the reader reconnecting instead of
+ // failing the in-flight read), so the compaction future may stay pending for far longer than the deletion
+ // can wait. Proceed with the cursor deletion right away in that case so a forced topic/namespace deletion
+ // cannot hang (issue #24148).
+ CompletableFuture<Long> compactionToWait =
+ isClosingOrDeleting ? CompletableFuture.<Long>completedFuture(null) : currentCompaction;
+ compactionToWait.exceptionally(compactionEx -> {
log.info("[{}][{}] Last compaction task failed, proceeding to delete the compaction cursor",
topic, subscriptionName, compactionEx);
return null;
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
index 1bc88ed0532d..940ec5dfe601 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
@@ -41,6 +41,7 @@ import org.apache.pulsar.common.api.proto.MessageIdData;
import org.apache.pulsar.common.api.proto.MessageMetadata;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.common.util.collections.GrowableArrayBlockingQueue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -160,6 +161,25 @@ public class RawReaderImpl implements RawReader {
return super.isUnrecoverableError(t);
}
+ @Override
+ public boolean connectionFailed(PulsarClientException exception) {
+ // A compaction reader is created with retryOnRecoverableErrors=false. When the topic is fenced or
+ // deleted, a reconnect can fail at the lookup/connection stage with a retriable error such as
+ // ServiceNotReadyException. The base ConsumerImpl.connectionFailed only consults isUnrecoverableError
+ // for non-retriable errors (or after the lookup deadline has passed), so for such a retriable error it
+ // would keep reconnecting and never fail the in-flight read, leaving the compaction future pending.
+ // Honor isUnrecoverableError here too so the reader is closed promptly and pending reads are failed,
+ // mirroring the subscribe-stage handling in ConsumerImpl.connectionOpened(). This matters for
+ // compaction: a never-failing read keeps the compaction future pending, which blocks forced
+ // topic/namespace deletion (issue #24148).
+ Throwable actError = FutureUtil.unwrapCompletionException(exception);
+ if (isUnrecoverableError(actError)) {
+ closeWhenReceivedUnrecoverableError(actError, null);
+ return false;
+ }
+ return super.connectionFailed(exception);
+ }
+
void tryCompletePending() {
CompletableFuture<RawMessage> future = null;
RawMessageAndCnx messageAndCnx = null;
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
index 9320e22e42e8..a043f71923a6 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
@@ -51,6 +51,7 @@ import org.apache.pulsar.broker.BrokerTestUtil;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
import org.apache.pulsar.broker.service.persistent.PersistentTopic;
import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.MessageRoutingMode;
import org.apache.pulsar.client.api.Producer;
@@ -643,6 +644,68 @@ public class RawReaderTest extends MockedPulsarServiceBaseTest {
assertTrue(readFuture.isCompletedExceptionally());
}
+ @Test(timeOut = 30000)
+ public void testConnectionFailureTerminatesReadWhenNotRetryingRecoverableErrors() throws Exception {
+ String topic = "persistent://my-property/my-ns/" + BrokerTestUtil.newUniqueName("reader");
+ admin.topics().createNonPartitionedTopic(topic);
+
+ // A compaction reader is created with retryOnRecoverableErrors=false. When the topic is fenced or deleted,
+ // a reconnect can fail at the lookup/connection stage (handled by ConsumerImpl.connectionFailed) with a
+ // retriable error such as ServiceNotReadyException. The base class would keep reconnecting, leaving the
+ // in-flight read pending forever; for a compaction reader that keeps the compaction future pending and
+ // blocks forced topic/namespace deletion (issue #24148). Such an unrecoverable error must instead
+ // terminate the reader and fail the in-flight read promptly.
+ ConsumerConfigurationData<byte[]> conf = new ConsumerConfigurationData<>();
+ conf.getTopicNames().add(topic);
+ conf.setSubscriptionName(subscription);
+ conf.setSubscriptionType(SubscriptionType.Exclusive);
+ conf.setReceiverQueueSize(DEFAULT_RECEIVER_QUEUE_SIZE);
+ conf.setSubscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
+ conf.setReadCompacted(true);
+ CompletableFuture<Consumer<byte[]>> consumerFuture = new CompletableFuture<>();
+ RawReaderImpl.RawConsumerImpl consumer = new RawReaderImpl.RawConsumerImpl(
+ (PulsarClientImpl) pulsarClient, conf, consumerFuture, false, false);
+ consumerFuture.get(10, TimeUnit.SECONDS);
+
+ CompletableFuture<RawMessage> readFuture = consumer.receiveRawAsync();
+ boolean keepReconnecting =
+ consumer.connectionFailed(new PulsarClientException.ServiceNotReadyException("injected"));
+
+ Assert.assertFalse(keepReconnecting);
+ Awaitility.await().atMost(10, TimeUnit.SECONDS).untilAsserted(() -> assertTrue(readFuture.isDone()));
+ assertTrue(readFuture.isCompletedExceptionally());
+ }
+
+ @Test(timeOut = 30000)
+ public void testConnectionFailureBeforeSubscribeFailsReaderCreation() throws Exception {
+ String topic = "persistent://my-property/my-ns/" + BrokerTestUtil.newUniqueName("reader");
+ admin.topics().createNonPartitionedTopic(topic);
+
+ // Same compaction-reader scenario as the test above, but the unrecoverable error (e.g. a lookup
+ // failure with ServiceNotReadyException) arrives BEFORE the initial subscribe completes. The
+ // subscribe (consumer) future must then be completed exceptionally; otherwise RawReader.create(...)
+ // would stay pending forever, which keeps the compaction future pending and blocks forced
+ // topic/namespace deletion (issue #24148).
+ ConsumerConfigurationData<byte[]> conf = new ConsumerConfigurationData<>();
+ conf.getTopicNames().add(topic);
+ conf.setSubscriptionName(subscription);
+ conf.setSubscriptionType(SubscriptionType.Exclusive);
+ conf.setReceiverQueueSize(DEFAULT_RECEIVER_QUEUE_SIZE);
+ conf.setSubscriptionInitialPosition(SubscriptionInitialPosition.Earliest);
+ conf.setReadCompacted(true);
+ CompletableFuture<Consumer<byte[]>> consumerFuture = new CompletableFuture<>();
+ RawReaderImpl.RawConsumerImpl consumer = new RawReaderImpl.RawConsumerImpl(
+ (PulsarClientImpl) pulsarClient, conf, consumerFuture, false, false);
+ // Inject the failure without waiting for the subscribe to complete, i.e. while the consumer
+ // future is still pending (construction returns before the async subscribe round-trip finishes).
+ boolean keepReconnecting =
+ consumer.connectionFailed(new PulsarClientException.ServiceNotReadyException("injected"));
+
+ Assert.assertFalse(keepReconnecting);
+ Awaitility.await().atMost(10, TimeUnit.SECONDS).untilAsserted(() -> assertTrue(consumerFuture.isDone()));
+ assertTrue(consumerFuture.isCompletedExceptionally());
+ }
+
@Test(timeOut = 100000)
public void testPauseAndResume() throws Exception {
log.info("-- Starting testPauseAndResume test --");
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index 6ce5780f623e..ac236c4f0770 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -2547,6 +2547,43 @@ public class CompactionTest extends MockedPulsarServiceBaseTest {
deleteTopicFuture.get(15, TimeUnit.SECONDS);
}
+ @Test(timeOut = 60 * 1000)
+ public void testForcedDeleteCompletesWhileCompactionStuck() throws Exception {
+ final String topicName = newUniqueName("persistent://my-tenant/my-ns/forced-delete-stuck-compaction");
+ admin.topics().createNonPartitionedTopic(topicName);
+ try (Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic(topicName).create()) {
+ for (int i = 0; i < 10; i++) {
+ producer.newMessage().key("key" + (i % 2)).value("value-" + i).send();
+ }
+ }
+
+ // Block the compaction at the phase-two seek so its future never completes on its own. This reproduces an
+ // in-flight compaction whose reader does not fail promptly when the topic is fenced (e.g. because a
+ // reconnect keeps retrying a retriable lookup-stage error): the compaction future stays pending (issue
+ // #24148).
+ CompletableFuture<Void> blockedSeek = new CompletableFuture<>();
+ CountDownLatch reachedSeek = new CountDownLatch(1);
+ AbstractTwoPhaseCompactor.injectionPhaseTwoSeek = (reader, id) -> {
+ reachedSeek.countDown();
+ return blockedSeek;
+ };
+ try {
+ PersistentTopic persistentTopic =
+ (PersistentTopic) pulsar.getBrokerService().getTopic(topicName, false).join().get();
+ persistentTopic.triggerCompaction();
+ assertTrue(reachedSeek.await(30, TimeUnit.SECONDS));
+ assertEquals(persistentTopic.compactionStatus().status, LongRunningProcessStatus.Status.RUNNING);
+
+ // The compaction future is stuck, but a forced deletion must complete promptly instead of waiting for
+ // the in-flight compaction to finish (issue #24148).
+ persistentTopic.deleteForcefully().get(15, TimeUnit.SECONDS);
+ } finally {
+ AbstractTwoPhaseCompactor.injectionPhaseTwoSeek = RawReader::seekAsync;
+ // Unblock the stuck compaction so it can unwind and release its reader.
+ blockedSeek.complete(null);
+ }
+ }
+
@Test
public void testForcedDeleteSucceedsAfterFailedCompaction() throws Exception {
String topicName = newUniqueName("persistent://my-tenant/my-ns/delete-after-failed-compaction");
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index 1038d84d89cf..64c67dec0718 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -1030,6 +1030,10 @@ public class ConsumerImpl<T> extends ConsumerBase<T> implements ConnectionHandle
final String cnxStr = cnx == null ? "null" : String.valueOf(cnx.channel().remoteAddress());
log.warn("[{}][{}] {} Closed consumer because get an error that does not support to retry: {} {}",
topic, subscription, cnxStr, t.getClass().getName(), t.getMessage());
+ // If the unrecoverable error occurs before the initial subscribe completes, fail the subscribe
+ // future as well; otherwise callers waiting on it (e.g. RawReader.create() / subscribeAsync())
+ // would hang forever. This is a no-op when the subscribe future has already completed.
+ subscribeFuture.completeExceptionally(t);
closeAsync().whenComplete((__, ex) -> {
if (ex == null) {
fail(t);
(pulsar) 09/17: [fix][broker] Run the message expiry check off the topic policy update path (#26040)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit ba47565904e6f1336c431f106d3efbf5166b6caa
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Thu Jun 18 02:09:04 2026 +0300
[fix][broker] Run the message expiry check off the topic policy update path (#26040)
(cherry picked from commit 476312837b5660b23b5ea2c6ad112bdfba88f2ca)
---
.../pulsar/broker/service/BrokerService.java | 1 +
.../broker/service/persistent/PersistentTopic.java | 26 ++++++++++++++++++++--
.../pulsar/broker/service/MessageTTLTest.java | 7 +++---
3 files changed, 29 insertions(+), 5 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index b67b0314c7e4..28647a79696b 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -287,6 +287,7 @@ public class BrokerService implements Closeable {
.register();
private final SingleThreadNonConcurrentFixedRateScheduler inactivityMonitor;
+ @Getter
private final SingleThreadNonConcurrentFixedRateScheduler messageExpiryMonitor;
private final SingleThreadNonConcurrentFixedRateScheduler compactionMonitor;
private final SingleThreadNonConcurrentFixedRateScheduler consumedLedgersMonitor;
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index c8f6e9aef93f..dc379fa9d96c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -225,6 +225,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
private final TopicName shadowSourceTopic;
public static final String DEDUPLICATION_CURSOR_NAME = "pulsar.dedup";
+ private volatile int lastMessageTtlInSeconds;
public static boolean isDedupCursorName(String name) {
return DEDUPLICATION_CURSOR_NAME.equals(name);
@@ -3738,8 +3739,9 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
});
producers.values().forEach(producer -> applyPoliciesFutureList.add(
producer.checkPermissionsAsync().thenRun(producer::checkEncryption)));
- // Check message expiry.
- applyPoliciesFutureList.add(FutureUtil.runWithCurrentThread(() -> checkMessageExpiry()));
+
+ // Check message expiry in the background so that it doesn't block updating or loading topic policies.
+ maybeCheckMessageExpiryOnPolicyUpdateInBackground();
// Update rate limiters.
applyPoliciesFutureList.add(FutureUtil.runWithCurrentThread(() -> updateDispatchRateLimiter()));
@@ -3764,6 +3766,26 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return applyPoliciesFutureList;
}
+ /**
+ * Triggers a message-expiry check on a policy update, but only when the message TTL actually changed, and
+ * runs it on the messageExpiryMonitor thread (the same thread the periodic expiry check uses) so it never
+ * blocks updating or loading topic policies. Unlike before, the expiry check is no longer part of the
+ * returned {@link #applyUpdatedTopicPolicies()} futures, so callers no longer wait for it; the periodic
+ * monitor provides eventual coverage and the check is idempotent.
+ */
+ private void maybeCheckMessageExpiryOnPolicyUpdateInBackground() {
+ int messageTtlInSeconds = topicPolicies.getMessageTTLInSeconds().get();
+ // don't check if the message expiry hasn't changed
+ if (messageTtlInSeconds > 0) {
+ if (lastMessageTtlInSeconds != messageTtlInSeconds) {
+ lastMessageTtlInSeconds = messageTtlInSeconds;
+ brokerService.getMessageExpiryMonitor().execute(this::checkMessageExpiry);
+ }
+ } else {
+ lastMessageTtlInSeconds = messageTtlInSeconds;
+ }
+ }
+
/**
*
* @return Backlog quota for topic
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/MessageTTLTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/MessageTTLTest.java
index beb054d86f5e..699f5638aabd 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/MessageTTLTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/MessageTTLTest.java
@@ -19,7 +19,7 @@
package org.apache.pulsar.broker.service;
import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.timeout;
import static org.mockito.Mockito.verify;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
@@ -138,12 +138,13 @@ public class MessageTTLTest extends BrokerTestBase {
Policies policies = admin.namespaces().getPolicies(namespace);
policies.message_ttl_in_seconds = 10;
topicRefMock.onPoliciesUpdate(policies);
- verify(topicRefMock, times(1)).checkMessageExpiry();
+ // checkMessageExpiry now runs asynchronously on the messageExpiryMonitor thread, so wait for it.
+ verify(topicRefMock, timeout(5000).times(1)).checkMessageExpiry();
TopicPolicies topicPolicies = new TopicPolicies();
topicPolicies.setMessageTTLInSeconds(5);
topicRefMock.onUpdate(topicPolicies);
- verify(topicRefMock, times(2)).checkMessageExpiry();
+ verify(topicRefMock, timeout(5000).times(2)).checkMessageExpiry();
}
@Test
(pulsar) 08/17: [fix][broker] Run topic policy notifications on the topic-ordered executor (#26042)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit b1dec33d5614e8b5ecdd989a7a1cfc7ce15e20d0
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Thu Jun 18 01:57:06 2026 +0300
[fix][broker] Run topic policy notifications on the topic-ordered executor (#26042)
(cherry picked from commit a9278c2c4c26deb0d2a1f6bb02bd0001e65c13a4)
---
.../pulsar/broker/service/AbstractTopic.java | 11 +++
.../pulsar/broker/service/BrokerService.java | 45 +++++++++--
.../SystemTopicBasedTopicPoliciesService.java | 93 ++++++++++++++--------
.../service/nonpersistent/NonPersistentTopic.java | 4 +-
.../broker/service/persistent/PersistentTopic.java | 7 +-
.../SystemTopicBasedTopicPoliciesServiceTest.java | 30 +++++++
6 files changed, 148 insertions(+), 42 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 0190c739a9c6..2a950c265918 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -42,6 +42,7 @@ import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import java.util.concurrent.atomic.AtomicLongFieldUpdater;
@@ -182,10 +183,16 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
protected final Clock clock;
protected Set<String> additionalSystemCursorNames = new TreeSet<>();
+ private final ExecutorService topicPoliciesNotifyThread;
public AbstractTopic(String topic, BrokerService brokerService) {
this.topic = topic;
this.namespace = TopicName.get(topic).getNamespaceObject();
+ // Pin the per-topic policies-notify thread once. BrokerService#getTopicPoliciesNotifyThread centralizes
+ // the topic-to-thread mapping so it stays consistent with SystemTopicBasedTopicPoliciesService. In unit
+ // tests that construct topics with a mock BrokerService this returns null (the thread is unused there).
+ this.topicPoliciesNotifyThread =
+ brokerService.getTopicPoliciesNotifyThread(TopicName.getPartitionedTopicName(topic));
this.clock = brokerService.getClock();
this.brokerService = brokerService;
this.producers = new ConcurrentHashMap<>();
@@ -1385,4 +1392,8 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
return COMPACTION_SUBSCRIPTION.equals(sub)
|| (additionalSystemCursorNames != null && additionalSystemCursorNames.contains(sub));
}
+
+ protected ExecutorService getPoliciesNotifyThread() {
+ return topicPoliciesNotifyThread;
+ }
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
index 37471948a90d..b67b0314c7e4 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
@@ -67,6 +67,7 @@ import java.util.concurrent.CompletionException;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.Semaphore;
@@ -2628,9 +2629,10 @@ public class BrokerService implements Closeable {
log.debug("Notifying topic that local policies have changed: {}", name);
}
topic.ifPresent(t -> {
- if (t instanceof PersistentTopic) {
- PersistentTopic topic1 = (PersistentTopic) t;
- topic1.onLocalPoliciesUpdate();
+ if (t instanceof PersistentTopic persistentTopic) {
+ runOnTopicPoliciesNotifyThread(t, () -> {
+ persistentTopic.onLocalPoliciesUpdate();
+ });
}
});
});
@@ -2650,7 +2652,8 @@ public class BrokerService implements Closeable {
log.info("[{}] updating with {}", namespace, policies);
topics.forEach((name, topicFuture) -> {
- if (namespace.includes(TopicName.get(name))) {
+ TopicName topicName = TopicName.get(name);
+ if (namespace.includes(topicName)) {
// If the topic is already created, immediately apply the updated policies, otherwise
// once the topic is created it'll apply the policies update
topicFuture.thenAccept(topic -> {
@@ -2658,7 +2661,11 @@ public class BrokerService implements Closeable {
log.debug("Notifying topic that policies have changed: {}", name);
}
- topic.ifPresent(t -> t.onPoliciesUpdate(policies));
+ topic.ifPresent(t -> {
+ runOnTopicPoliciesNotifyThread(t, () -> {
+ t.onPoliciesUpdate(policies);
+ });
+ });
});
}
});
@@ -2669,6 +2676,16 @@ public class BrokerService implements Closeable {
}, pulsar.getExecutor());
}
+ private void runOnTopicPoliciesNotifyThread(Topic t, Runnable runnable) {
+ ExecutorService policiesNotifyThread;
+ if (t instanceof AbstractTopic abstractTopic) {
+ policiesNotifyThread = abstractTopic.getPoliciesNotifyThread();
+ } else {
+ policiesNotifyThread = getTopicPoliciesNotifyThread(TopicName.getPartitionedTopicName(t.getName()));
+ }
+ policiesNotifyThread.execute(runnable);
+ }
+
private void handleDynamicConfigurationUpdates() {
DynamicConfigurationResources dynamicConfigResources = null;
try {
@@ -3516,6 +3533,24 @@ public class BrokerService implements Closeable {
return topicOrderedExecutor;
}
+ /**
+ * Returns the single executor thread used to apply topic-policy updates for the given topic. All
+ * topic-policy notifications and policy application for a topic must run on this deterministically-chosen
+ * thread so that they are serialized and never run concurrently. Centralizing the topic-to-thread mapping
+ * here keeps it consistent between {@link AbstractTopic} and
+ * {@link SystemTopicBasedTopicPoliciesService} so the two cannot accidentally diverge.
+ */
+ public ExecutorService getTopicPoliciesNotifyThread(TopicName topicName) {
+ TopicName baseTopicName;
+ if (topicName.isPartitioned()) {
+ // for partitioned topics, we need to use the base topic name
+ baseTopicName = TopicName.get(topicName.getPartitionedTopicName());
+ } else {
+ baseTopicName = topicName;
+ }
+ return topicOrderedExecutor.chooseThread(baseTopicName);
+ }
+
/**
* If per-broker unacked message reached to limit then it blocks dispatcher if its unacked message limit has been
* reached to {@link #maxUnackedMsgsPerDispatcher}.
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
index 7e1b453b35b8..623933eaafcb 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
@@ -35,6 +35,7 @@ import java.util.PriorityQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
@@ -485,16 +486,7 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
if (msg.getValue() == null) {
TopicName topicName = TopicName.get(TopicPoliciesService.unwrapEventKey(msg.getKey())
.getPartitionedTopicName());
- List<TopicPolicyListener> listeners = this.listeners.get(topicName);
- if (listeners != null) {
- for (TopicPolicyListener listener : listeners) {
- try {
- listener.onUpdate(null);
- } catch (Throwable error) {
- log.error("[{}] call listener error.", topicName, error);
- }
- }
- }
+ notifyListenersForTopic(topicName, null);
return;
}
@@ -504,19 +496,62 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
TopicPoliciesEvent event = msg.getValue().getTopicPoliciesEvent();
TopicName topicName = TopicName.get(event.getDomain(), event.getTenant(),
event.getNamespace(), event.getTopic());
- List<TopicPolicyListener> listeners = this.listeners.get(topicName);
- if (listeners != null) {
- TopicPolicies policies = event.getPolicies();
- for (TopicPolicyListener listener : listeners) {
- try {
- listener.onUpdate(policies);
- } catch (Throwable error) {
- log.error("[{}] call listener error.", topicName, error);
- }
+ notifyListenersForTopic(topicName, event.getPolicies());
+ }
+
+ /**
+ * Notifies the topic-policy listeners registered for {@code topicName} of a policy update.
+ *
+ * <p>The {@link TopicPolicyListener#onUpdate} calls are dispatched to the per-topic ordered executor
+ * rather than run inline. The reader callbacks that drive notifications (the {@link #initPolicesCache}
+ * replay loop and {@link #readMorePoliciesAsync}) run on the single, process-wide shared
+ * {@code broker-client-shared-internal-executor} thread. A listener (e.g.
+ * {@code PersistentTopic.onUpdate} -> {@code applyUpdatedTopicPolicies}) can perform non-trivial and
+ * even blocking work, so running it inline serializes and can stall topic-policy loading for every
+ * namespace (issue #26037). Keying {@code executeOrdered} by {@code topicName} preserves per-topic
+ * notification ordering.
+ */
+ private void notifyListenersForTopic(TopicName topicName, @Nullable TopicPolicies policies) {
+ // The per-topic value is a CopyOnWriteArrayList, so iterating it later on the executor thread stays
+ // safe even if a listener is registered/unregistered between dispatch and execution.
+ List<TopicPolicyListener> topicListeners = listeners.get(topicName);
+ if (topicListeners == null || topicListeners.isEmpty()) {
+ return;
+ }
+ pulsarService.getBrokerService().getTopicPoliciesNotifyThread(topicName).execute(() -> {
+ internalNotifyTopicListeners(topicName, policies, topicListeners);
+ });
+ }
+
+ // this method should only be called from the topic ordered executor thread
+ // use notifyListenersForTopic/notifyListenersForTopicAsync instead
+ private static void internalNotifyTopicListeners(TopicName topicName, @Nullable TopicPolicies policies,
+ List<TopicPolicyListener> topicListeners) {
+ for (TopicPolicyListener listener : topicListeners) {
+ try {
+ listener.onUpdate(policies);
+ } catch (Throwable error) {
+ log.error("[{}] Error in notifying listener {} on topic policy update.",
+ topicName, listener, error);
}
}
}
+ private CompletableFuture<Void> notifyListenersForTopicAsync(TopicName topicName,
+ @Nullable TopicPolicies policies) {
+ // The per-topic value is a CopyOnWriteArrayList, so iterating it later on the executor thread stays
+ // safe even if a listener is registered/unregistered between dispatch and execution.
+ List<TopicPolicyListener> topicListeners = listeners.get(topicName);
+ if (topicListeners == null || topicListeners.isEmpty()) {
+ return CompletableFuture.completedFuture(null);
+ }
+ ExecutorService pinnedTopicOrderedExecutor =
+ pulsarService.getBrokerService().getTopicPoliciesNotifyThread(topicName);
+ return CompletableFuture.runAsync(() -> {
+ internalNotifyTopicListeners(topicName, policies, topicListeners);
+ }, pinnedTopicOrderedExecutor);
+ }
+
@Override
public CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsync(TopicName topicName, GetType type) {
requireNonNull(topicName);
@@ -827,19 +862,13 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
}
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener listener : listeners.get(topicName)) {
- try {
- listener.onUpdate(topicPolicies);
- } catch (Throwable error) {
- log.error("[{}] call listener error.", topicName, error);
- }
- }
- }
- }));
-
- future.complete(null);
+ List<CompletableFuture<Void>> notifyFutures = new ArrayList<>();
+ for (Map.Entry<TopicName, TopicPolicies> entry : policiesCache.entrySet()) {
+ TopicName topicName = entry.getKey();
+ TopicPolicies policies = entry.getValue();
+ notifyFutures.add(notifyListenersForTopicAsync(topicName, policies));
+ }
+ FutureUtil.completeAfter(future, FutureUtil.waitForAll(notifyFutures));
}
});
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index e79c642e7b75..4e04fd268155 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -164,7 +164,7 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
public CompletableFuture<Void> initialize() {
return brokerService.pulsar().getPulsarResources().getNamespaceResources()
.getPoliciesAsync(TopicName.get(topic).getNamespaceObject())
- .thenCompose(optPolicies -> {
+ .thenComposeAsync(optPolicies -> {
final Policies policies;
if (optPolicies.isEmpty()) {
log.warn("[{}] Policies not present and isEncryptionRequired will be set to false", topic);
@@ -179,7 +179,7 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
updatePublishRateLimiter();
updateResourceGroupLimiter(policies);
return updateClusterMigrated();
- });
+ }, getPoliciesNotifyThread());
}
@Override
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index d68e21740502..c8f6e9aef93f 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -495,7 +495,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
this.isEncryptionRequired = policies.encryption_required;
isAllowAutoUpdateSchema = policies.is_allow_auto_update_schema;
- }, getOrderedExecutor())
+ }, getPoliciesNotifyThread())
.thenCompose(ignore -> initTopicPolicy())
.thenCompose(ignore -> removeOrphanReplicationCursors())
.exceptionally(ex -> {
@@ -4733,6 +4733,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
protected CompletableFuture<Void> initTopicPolicy() {
final var topicPoliciesService = brokerService.pulsar().getTopicPoliciesService();
final var partitionedTopicName = TopicName.getPartitionedTopicName(topic);
+
return topicPoliciesService.registerListenerAsync(partitionedTopicName, this).thenCompose(registered -> {
if (!registered) {
return CompletableFuture.completedFuture(null);
@@ -4743,11 +4744,11 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
TopicPoliciesService.GetType.GLOBAL_ONLY)
.thenAcceptAsync(optionalPolicies -> optionalPolicies.ifPresent(this::onUpdate),
- brokerService.getTopicOrderedExecutor())
+ getPoliciesNotifyThread())
.thenCompose(__ -> topicPoliciesService.getTopicPoliciesAsync(partitionedTopicName,
TopicPoliciesService.GetType.LOCAL_ONLY))
.thenAcceptAsync(optionalPolicies -> optionalPolicies.ifPresent(this::onUpdate),
- brokerService.getTopicOrderedExecutor());
+ getPoliciesNotifyThread());
});
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
index 1b5182a45452..dbce03b9d1bb 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
@@ -134,6 +134,36 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
Assert.assertFalse(systemTopicBasedTopicPoliciesService.listeners.containsKey(topicName));
}
+ @Test
+ public void testListenerNotificationRunsOffSharedReaderThread() throws Exception {
+ // Regression test for #26037: topic-policy listener callbacks must not run on the single,
+ // process-wide shared "broker-client-shared-internal-executor" reader thread. A slow or blocking
+ // onUpdate there serializes — and can stall — topic-policy loading for every namespace. The
+ // notification must instead be dispatched to the per-topic ordered executor ("broker-topic-workers").
+
+ // Initialize the policy cache and start the change-events reader for the namespace.
+ systemTopicBasedTopicPoliciesService.updateTopicPoliciesAsync(TOPIC1, false, false, topicPolicies ->
+ topicPolicies.setMaxConsumerPerTopic(10)).get();
+ Awaitility.await().untilAsserted(() -> Assert.assertTrue(systemTopicBasedTopicPoliciesService
+ .getPoliciesCacheInit(TOPIC1.getNamespaceObject()).isDone()));
+
+ // Register a listener that records the thread its onUpdate runs on.
+ CompletableFuture<String> onUpdateThreadName = new CompletableFuture<>();
+ TopicPolicyListener listener = data -> onUpdateThreadName.complete(Thread.currentThread().getName());
+ systemTopicBasedTopicPoliciesService.registerListenerAsync(TOPIC1, listener).get();
+
+ // A live policy update flows through readMorePoliciesAsync -> notifyListener -> listener.onUpdate.
+ systemTopicBasedTopicPoliciesService.updateTopicPoliciesAsync(TOPIC1, false, false, topicPolicies ->
+ topicPolicies.setMaxConsumerPerTopic(20)).get();
+
+ String threadName = onUpdateThreadName.get(30, TimeUnit.SECONDS);
+ assertFalse("listener.onUpdate must not run on the shared broker-client reader thread, but ran on: "
+ + threadName,
+ threadName.contains("broker-client-shared-internal-executor"));
+ assertTrue("listener.onUpdate should run on the per-topic ordered executor, but ran on: " + threadName,
+ threadName.contains("broker-topic-workers"));
+ }
+
@Test
public void testGetPolicy() throws Exception {
(pulsar) 17/17: [fix] functions: Run worker leader-election off the consumer event-listener thread (#26059)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 95ce1733b62365d66956074b8e933eb032b95199
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Sat Jun 20 14:00:29 2026 -0700
[fix] functions: Run worker leader-election off the consumer event-listener thread (#26059)
(cherry picked from commit ced1280a108709b9fddbb4f8a161fd288c3e2565)
---
.../pulsar/functions/worker/LeaderService.java | 28 +++++++++++++++++++++-
.../pulsar/functions/worker/LeaderServiceTest.java | 8 +++++++
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/LeaderService.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/LeaderService.java
index e7816f06aacc..d15b91746355 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/LeaderService.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/LeaderService.java
@@ -18,6 +18,11 @@
*/
package org.apache.pulsar.functions.worker;
+import com.google.common.annotations.VisibleForTesting;
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
import java.util.function.Supplier;
import lombok.extern.slf4j.Slf4j;
import org.apache.pulsar.client.api.Consumer;
@@ -42,6 +47,11 @@ public class LeaderService implements AutoCloseable, ConsumerEventListener {
private final WorkerConfig workerConfig;
private final PulsarClient pulsarClient;
private volatile boolean isLeader = false;
+ // The consumer event listener callbacks (becameActive/becameInactive) run the blocking
+ // leader-election routines on this dedicated single-threaded executor so that the Pulsar client's
+ // shared consumer-listener thread is not blocked. The single thread also preserves event ordering.
+ private final ExecutorService executor =
+ Executors.newSingleThreadExecutor(new DefaultThreadFactory("function-worker-leader"));
static final String COORDINATION_TOPIC_SUBSCRIPTION = "participants";
@@ -89,6 +99,12 @@ public class LeaderService implements AutoCloseable, ConsumerEventListener {
@Override
public void becameActive(Consumer<?> consumer, int partitionId) {
+ // Run the (blocking) become-leader routine on a dedicated executor so the consumer
+ // event-listener thread, which is shared with the Pulsar client, is not blocked.
+ executor.execute(() -> becameActiveInternal(consumer, partitionId));
+ }
+
+ private void becameActiveInternal(Consumer<?> consumer, int partitionId) {
synchronized (this) {
if (isLeader) {
return;
@@ -148,7 +164,11 @@ public class LeaderService implements AutoCloseable, ConsumerEventListener {
}
@Override
- public synchronized void becameInactive(Consumer<?> consumer, int partitionId) {
+ public void becameInactive(Consumer<?> consumer, int partitionId) {
+ executor.execute(() -> becameInactiveInternal(consumer, partitionId));
+ }
+
+ private synchronized void becameInactiveInternal(Consumer<?> consumer, int partitionId) {
if (isLeader) {
log.info("Worker {} lost the leadership.", consumerName);
isLeader = false;
@@ -176,10 +196,16 @@ public class LeaderService implements AutoCloseable, ConsumerEventListener {
return isLeader;
}
+ @VisibleForTesting
+ void joinPendingEventTasks() throws InterruptedException, ExecutionException {
+ executor.submit(() -> { }).get();
+ }
+
@Override
public void close() throws PulsarClientException {
if (consumer != null) {
consumer.close();
}
+ executor.shutdown();
}
}
diff --git a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/LeaderServiceTest.java b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/LeaderServiceTest.java
index 5c10a59bd138..03cbd772ab2e 100644
--- a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/LeaderServiceTest.java
+++ b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/LeaderServiceTest.java
@@ -135,6 +135,7 @@ public class LeaderServiceTest {
verify(mockClient, times(1)).newConsumer();
listenerHolder.get().becameActive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertTrue(leaderService.isLeader());
verify(functionMetadataManager, times(1)).getIsInitialized();
@@ -150,6 +151,7 @@ public class LeaderServiceTest {
verify(schedulerManager, times((1))).initialize(any());
listenerHolder.get().becameInactive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertFalse(leaderService.isLeader());
verify(functionAssignmentTailer, times(1)).startFromMessage(messageId);
@@ -165,6 +167,7 @@ public class LeaderServiceTest {
verify(mockClient, times(1)).newConsumer();
listenerHolder.get().becameActive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertTrue(leaderService.isLeader());
verify(functionMetadataManager, times(1)).acquireExclusiveWrite(any());
@@ -176,6 +179,7 @@ public class LeaderServiceTest {
verify(schedulerManager, times((1))).initialize(any());
listenerHolder.get().becameInactive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertFalse(leaderService.isLeader());
verify(functionAssignmentTailer, times(1)).start();
@@ -195,6 +199,7 @@ public class LeaderServiceTest {
when(schedulerManager.acquireExclusiveWrite(any())).thenThrow(new WorkerUtils.NotLeaderAnymore());
listenerHolder.get().becameActive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
// should have failed to become leader
assertFalse(leaderService.isLeader());
@@ -211,6 +216,7 @@ public class LeaderServiceTest {
verify(schedulerManager, times((0))).initialize(any());
listenerHolder.get().becameInactive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertFalse(leaderService.isLeader());
verify(functionAssignmentTailer, times(0)).startFromMessage(messageId);
@@ -231,6 +237,7 @@ public class LeaderServiceTest {
when(functionMetadataManager.acquireExclusiveWrite(any())).thenThrow(new WorkerUtils.NotLeaderAnymore());
listenerHolder.get().becameActive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
// should have failed to become leader
assertFalse(leaderService.isLeader());
@@ -247,6 +254,7 @@ public class LeaderServiceTest {
verify(schedulerManager, times((0))).initialize(any());
listenerHolder.get().becameInactive(mockConsumer, 0);
+ leaderService.joinPendingEventTasks();
assertFalse(leaderService.isLeader());
verify(functionAssignmentTailer, times(0)).startFromMessage(messageId);
(pulsar) 12/17: [fix][broker] Avoid blocking metadata read on the IO thread when redirecting migrated producers/consumers (#26051)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a08f11b5ff9c64a13fa4f7fbc4f36e58b1c9f2b3
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Thu Jun 18 02:40:15 2026 -0700
[fix][broker] Avoid blocking metadata read on the IO thread when redirecting migrated producers/consumers (#26051)
(cherry picked from commit ac053c38fe40723bcee193c1aa6ab5e1d39d1779)
---
.../broker/service/AbstractBaseDispatcher.java | 3 +-
.../pulsar/broker/service/AbstractTopic.java | 14 +-
.../org/apache/pulsar/broker/service/Consumer.java | 22 ++-
.../apache/pulsar/broker/service/ServerCnx.java | 164 +++++++++++++--------
.../service/nonpersistent/NonPersistentTopic.java | 30 ++--
.../broker/service/persistent/PersistentTopic.java | 77 +++++-----
6 files changed, 174 insertions(+), 136 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java
index f074f234b873..b86bbc6dcb9d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java
@@ -469,7 +469,8 @@ public abstract class AbstractBaseDispatcher extends EntryFilterSupport implemen
public static void checkAndApplyReachedEndOfTopicOrTopicMigration(PersistentTopic topic, List<Consumer> consumers) {
if (topic.isMigrated()) {
- consumers.forEach(c -> c.topicMigrated(topic.getMigratedClusterUrl()));
+ topic.getMigratedClusterUrlAsync()
+ .thenAccept(clusterUrl -> consumers.forEach(c -> c.topicMigrated(clusterUrl)));
} else {
consumers.forEach(Consumer::reachedEndOfTopic);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
index 5dad5b1b0799..e36dbee95239 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java
@@ -1341,8 +1341,8 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
subscribeRateInBroker(brokerService.pulsar().getConfiguration()));
}
- public Optional<ClusterUrl> getMigratedClusterUrl() {
- return getMigratedClusterUrl(brokerService.getPulsar(), topic);
+ public CompletableFuture<Optional<ClusterUrl>> getMigratedClusterUrlAsync() {
+ return getMigratedClusterUrlAsync(brokerService.getPulsar(), topic);
}
public static CompletableFuture<Boolean> isClusterMigrationEnabled(PulsarService pulsar,
@@ -1382,16 +1382,6 @@ public abstract class AbstractTopic implements Topic, TopicPolicyListener {
.thenApply(policies -> policies.isPresent() && policies.get().migrated);
}
- public static Optional<ClusterUrl> getMigratedClusterUrl(PulsarService pulsar, String topic) {
- try {
- return getMigratedClusterUrlAsync(pulsar, topic)
- .get(pulsar.getPulsarResources().getClusterResources().getOperationTimeoutSec(), TimeUnit.SECONDS);
- } catch (Exception e) {
- log.warn("[{}] Failed to get migration cluster URL", topic, e);
- }
- return Optional.empty();
- }
-
public boolean isSystemCursor(String sub) {
return COMPACTION_SUBSCRIPTION.equals(sub)
|| (additionalSystemCursorNames != null && additionalSystemCursorNames.contains(sub));
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 1a49325ca801..e9ac85aaa2db 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -905,20 +905,16 @@ public class Consumer {
}
}
- public boolean checkAndApplyTopicMigration() {
- if (subscription.isSubscriptionMigrated()) {
- Optional<ClusterUrl> clusterUrl = AbstractTopic.getMigratedClusterUrl(cnx.getBrokerService().getPulsar(),
- topicName);
- if (clusterUrl.isPresent()) {
- ClusterUrl url = clusterUrl.get();
- cnx.getCommandSender().sendTopicMigrated(ResourceType.Consumer, consumerId, url.getBrokerServiceUrl(),
- url.getBrokerServiceUrlTls());
- // disconnect consumer after sending migrated cluster url
- disconnect();
- return true;
- }
+ public CompletableFuture<Boolean> checkAndApplyTopicMigrationAsync() {
+ if (!subscription.isSubscriptionMigrated()) {
+ return CompletableFuture.completedFuture(false);
}
- return false;
+ return AbstractTopic.getMigratedClusterUrlAsync(cnx.getBrokerService().getPulsar(), topicName)
+ .thenApply(clusterUrl -> {
+ // topicMigrated() sends the migrated cluster url and disconnects the consumer if present
+ topicMigrated(clusterUrl);
+ return clusterUrl.isPresent();
+ });
}
/**
* Checks if consumer-blocking on unAckedMessages is allowed for below conditions:<br/>
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
index 88bed34bf96f..43ca896d6bff 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@@ -26,7 +26,7 @@ import static org.apache.commons.lang3.StringUtils.isNotBlank;
import static org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.unsafeGetPartitionedTopicMetadataAsync;
import static org.apache.pulsar.broker.lookup.TopicLookupBase.lookupTopicAsync;
import static org.apache.pulsar.broker.service.ServerCnxThrottleTracker.ThrottleType;
-import static org.apache.pulsar.broker.service.persistent.PersistentTopic.getMigratedClusterUrl;
+import static org.apache.pulsar.broker.service.persistent.PersistentTopic.getMigratedClusterUrlAsync;
import static org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.ignoreUnrecoverableBKException;
import static org.apache.pulsar.common.api.proto.ProtocolVersion.v5;
import static org.apache.pulsar.common.naming.Constants.WEBSOCKET_DUMMY_ORIGINAL_PRINCIPLE;
@@ -1486,8 +1486,9 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
}
});
})
- .thenAcceptAsync(consumer -> {
- if (consumer.checkAndApplyTopicMigration()) {
+ .thenComposeAsync(consumer -> consumer.checkAndApplyTopicMigrationAsync()
+ .thenAcceptAsync(migrated -> {
+ if (migrated) {
log.info("[{}] Disconnecting consumer {} on migrated subscription on topic {} / {}",
remoteAddress, consumerId, subscriptionName, topicName);
consumers.remove(consumerId, consumerFuture);
@@ -1520,7 +1521,7 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
consumers.remove(consumerId, consumerFuture);
}
- }, ctx.executor())
+ }, ctx.executor()), ctx.executor())
.exceptionallyAsync(exception -> {
if (exception.getCause() instanceof ConsumerBusyException) {
if (log.isDebugEnabled()) {
@@ -1531,23 +1532,12 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
exception.getCause().getMessage());
}
} else if (exception.getCause() instanceof BrokerServiceException.TopicMigratedException) {
- Optional<ClusterUrl> clusterURL = getMigratedClusterUrl(service.getPulsar(),
- topicName.toString());
- if (clusterURL.isPresent()) {
- log.info("[{}] redirect migrated consumer to topic {}: "
- + "consumerId={}, subName={}, {}", remoteAddress,
- topicName, consumerId, subscriptionName, exception.getCause().getMessage());
- boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Consumer, consumerId,
- clusterURL.get().getBrokerServiceUrl(),
- clusterURL.get().getBrokerServiceUrlTls());
- if (!msgSent) {
- log.info("consumer client doesn't support topic migration handling {}-{}-{}",
- topicName, remoteAddress, consumerId);
- }
- consumers.remove(consumerId, consumerFuture);
- closeConsumer(consumerId, Optional.empty());
- return null;
- }
+ getMigratedClusterUrlAsync(service.getPulsar(), topicName.toString())
+ .exceptionally(e -> Optional.empty())
+ .thenAcceptAsync(clusterURL -> redirectOrFailMigratedConsumer(requestId,
+ consumerId, subscriptionName, topicName, consumerFuture, exception,
+ clusterURL), ctx.executor());
+ return null;
} else if (exception.getCause() instanceof BrokerServiceException) {
log.warn("[{}][{}][{}] Failed to create consumer: consumerId={}, {}",
remoteAddress, topicName, subscriptionName,
@@ -1585,6 +1575,89 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
}, ctx.executor());
}
+ private void redirectOrFailMigratedConsumer(long requestId, long consumerId, String subscriptionName,
+ TopicName topicName, CompletableFuture<Consumer> consumerFuture, Throwable exception,
+ Optional<ClusterUrl> clusterURL) {
+ if (clusterURL.isPresent()) {
+ log.info("[{}] redirect migrated consumer to topic {}: consumerId={}, subName={}, {}", remoteAddress,
+ topicName, consumerId, subscriptionName, exception.getCause().getMessage());
+ boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Consumer, consumerId,
+ clusterURL.get().getBrokerServiceUrl(),
+ clusterURL.get().getBrokerServiceUrlTls());
+ if (!msgSent) {
+ log.info("consumer client doesn't support topic migration handling {}-{}-{}",
+ topicName, remoteAddress, consumerId);
+ }
+ consumers.remove(consumerId, consumerFuture);
+ closeConsumer(consumerId, Optional.empty());
+ } else {
+ // If client timed out, the future would have been completed by subsequent close.
+ // Send error back to client, only if not completed already.
+ if (consumerFuture.completeExceptionally(exception)) {
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(exception.getCause()),
+ exception.getCause().getMessage());
+ }
+ consumers.remove(consumerId, consumerFuture);
+ }
+ }
+
+ private void redirectOrFailMigratedProducer(long requestId, long producerId, String producerName,
+ TopicName topicName, CompletableFuture<Producer> producerFuture, Throwable exception,
+ Optional<ClusterUrl> clusterURL) {
+ if (clusterURL.isPresent()) {
+ log.info("[{}] redirect migrated producer to topic {}: producerId={}, producerName = {}, {}",
+ remoteAddress, topicName, producerId, producerName, exception.getCause().getMessage());
+ boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Producer, producerId,
+ clusterURL.get().getBrokerServiceUrl(), clusterURL.get().getBrokerServiceUrlTls());
+ if (!msgSent) {
+ log.info("client doesn't support topic migration handling {}-{}-{}", topicName,
+ remoteAddress, producerId);
+ }
+ producers.remove(producerId, producerFuture);
+ closeProducer(producerId, -1L, Optional.empty());
+ } else {
+ log.error("[{}] Failed to create topic {}, producerId={}", remoteAddress, topicName, producerId,
+ exception);
+ if (producerFuture.completeExceptionally(exception)) {
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(exception.getCause()),
+ exception.getCause().getMessage());
+ }
+ producers.remove(producerId, producerFuture);
+ }
+ }
+
+ private void redirectOrFailMigratedProducerInQueue(long requestId, long producerId, String producerName,
+ TopicName topicName, Topic topic, Producer producer, CompletableFuture<Producer> producerFuture,
+ Throwable ex, Optional<ClusterUrl> clusterURL) {
+ if (clusterURL.isPresent() && topic.shouldProducerMigrate()) {
+ log.info("[{}] redirect migrated producer to topic {}: producerId={}, producerName = {}, {}",
+ remoteAddress, topicName, producerId, producerName, ex.getCause().getMessage());
+ boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Producer, producerId,
+ clusterURL.get().getBrokerServiceUrl(), clusterURL.get().getBrokerServiceUrlTls());
+ if (!msgSent) {
+ log.info("client doesn't support topic migration handling {}-{}-{}", topic,
+ remoteAddress, producerId);
+ }
+ closeProducer(producer);
+ } else {
+ if (clusterURL.isPresent()) {
+ log.info("Topic {} is migrated but replication backlog exist: "
+ + "producerId = {}, producerName = {}, {}", topicName,
+ producerId, producerName, ex.getCause().getMessage());
+ } else {
+ log.warn("[{}] failed producer because migration url not configured topic {}: producerId={}, {}",
+ remoteAddress, topicName, producerId, ex.getCause().getMessage());
+ }
+ producer.closeNow(true);
+ if (producerFuture.completeExceptionally(ex)) {
+ commandSender.sendErrorResponse(requestId,
+ BrokerServiceException.getClientErrorCode(ex), ex.getMessage());
+ }
+ }
+ }
+
private SchemaData getSchema(Schema protocolSchema) {
return SchemaData.builder()
.data(protocolSchema.getSchemaData())
@@ -1818,21 +1891,11 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
producers.remove(producerId, producerFuture);
return null;
} else if (cause instanceof BrokerServiceException.TopicMigratedException) {
- Optional<ClusterUrl> clusterURL = getMigratedClusterUrl(service.getPulsar(), topicName.toString());
- if (clusterURL.isPresent()) {
- log.info("[{}] redirect migrated producer to topic {}: "
- + "producerId={}, producerName = {}, {}", remoteAddress,
- topicName, producerId, producerName, cause.getMessage());
- boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Producer, producerId,
- clusterURL.get().getBrokerServiceUrl(), clusterURL.get().getBrokerServiceUrlTls());
- if (!msgSent) {
- log.info("client doesn't support topic migration handling {}-{}-{}", topicName,
- remoteAddress, producerId);
- }
- producers.remove(producerId, producerFuture);
- closeProducer(producerId, -1L, Optional.empty());
- return null;
- }
+ getMigratedClusterUrlAsync(service.getPulsar(), topicName.toString())
+ .exceptionally(e -> Optional.empty())
+ .thenAcceptAsync(clusterURL -> redirectOrFailMigratedProducer(requestId, producerId,
+ producerName, topicName, producerFuture, exception, clusterURL), ctx.executor());
+ return null;
}
// Do not print stack traces for expected exceptions
@@ -1918,29 +1981,12 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
producers.remove(producerId, producerFuture);
}, ctx.executor()).exceptionallyAsync(ex -> {
if (ex.getCause() instanceof BrokerServiceException.TopicMigratedException) {
- Optional<ClusterUrl> clusterURL = getMigratedClusterUrl(service.getPulsar(), topic.getName());
- if (clusterURL.isPresent()) {
- if (!topic.shouldProducerMigrate()) {
- log.info("Topic {} is migrated but replication backlog exist: "
- + "producerId = {}, producerName = {}, {}", topicName,
- producerId, producerName, ex.getCause().getMessage());
- } else {
- log.info("[{}] redirect migrated producer to topic {}: "
- + "producerId={}, producerName = {}, {}", remoteAddress,
- topicName, producerId, producerName, ex.getCause().getMessage());
- boolean msgSent = commandSender.sendTopicMigrated(ResourceType.Producer, producerId,
- clusterURL.get().getBrokerServiceUrl(), clusterURL.get().getBrokerServiceUrlTls());
- if (!msgSent) {
- log.info("client doesn't support topic migration handling {}-{}-{}", topic,
- remoteAddress, producerId);
- }
- closeProducer(producer);
- return null;
- }
- } else {
- log.warn("[{}] failed producer because migration url not configured topic {}: producerId={}, {}",
- remoteAddress, topicName, producerId, ex.getCause().getMessage());
- }
+ getMigratedClusterUrlAsync(service.getPulsar(), topic.getName())
+ .exceptionally(e -> Optional.empty())
+ .thenAcceptAsync(clusterURL -> redirectOrFailMigratedProducerInQueue(requestId, producerId,
+ producerName, topicName, topic, producer, producerFuture, ex, clusterURL),
+ ctx.executor());
+ return null;
} else if (ex.getCause() instanceof BrokerServiceException.ProducerFencedException) {
if (log.isDebugEnabled()) {
log.debug("[{}] Failed to add producer to topic {}: producerId={}, {}",
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
index 0e61bc059ab5..bedf96ca0406 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentTopic.java
@@ -81,7 +81,6 @@ import org.apache.pulsar.common.api.proto.KeySharedMeta;
import org.apache.pulsar.common.naming.SystemTopicNames;
import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.BacklogQuota;
-import org.apache.pulsar.common.policies.data.ClusterPolicies.ClusterUrl;
import org.apache.pulsar.common.policies.data.ManagedLedgerInternalStats.CursorStats;
import org.apache.pulsar.common.policies.data.PersistentTopicInternalStats;
import org.apache.pulsar.common.policies.data.Policies;
@@ -338,7 +337,7 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
false, cnx, cnx.getAuthRole(), metadata, readCompacted, keySharedMeta, MessageId.latest,
DEFAULT_CONSUMER_EPOCH, schemaType);
if (isMigrated()) {
- consumer.topicMigrated(getMigratedClusterUrl());
+ getMigratedClusterUrlAsync().thenAccept(consumer::topicMigrated);
}
addConsumerToSubscription(subscription, consumer).thenRun(() -> {
@@ -1015,20 +1014,21 @@ public class NonPersistentTopic extends AbstractTopic implements Topic, TopicPol
return CompletableFuture.completedFuture(null);
}
- Optional<ClusterUrl> url = getMigratedClusterUrl();
- if (url.isPresent()) {
- this.migrated = true;
- producers.forEach((__, producer) -> {
- producer.topicMigrated(url);
- });
- subscriptions.forEach((__, sub) -> {
- sub.getConsumers().forEach((consumer) -> {
- consumer.topicMigrated(url);
+ return getMigratedClusterUrlAsync().thenCompose(url -> {
+ if (url.isPresent()) {
+ this.migrated = true;
+ producers.forEach((__, producer) -> {
+ producer.topicMigrated(url);
});
- });
- return disconnectReplicators().thenCompose(__ -> checkAndUnsubscribeSubscriptions());
- }
- return CompletableFuture.completedFuture(null);
+ subscriptions.forEach((__, sub) -> {
+ sub.getConsumers().forEach((consumer) -> {
+ consumer.topicMigrated(url);
+ });
+ });
+ return disconnectReplicators().thenCompose(__ -> checkAndUnsubscribeSubscriptions());
+ }
+ return CompletableFuture.<Void>completedFuture(null);
+ });
}
private CompletableFuture<Void> checkAndUnsubscribeSubscriptions() {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index addcfff18a2d..36b02a0f8b22 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -778,18 +778,23 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
// close all producers
CompletableFuture<Void> disconnectProducersFuture;
if (producers.size() > 0) {
- List<CompletableFuture<Void>> futures = new ArrayList<>();
// send migration url metadata to producers before disconnecting them
- if (isMigrated()) {
- if (!shouldProducerMigrate()) {
+ CompletableFuture<Void> sendMigrationUrlFuture;
+ if (isMigrated() && shouldProducerMigrate()) {
+ sendMigrationUrlFuture = getMigratedClusterUrlAsync().thenAccept(clusterUrl ->
+ producers.forEach((__, producer) -> producer.topicMigrated(clusterUrl)));
+ } else {
+ if (isMigrated()) {
log.info("Topic {} is migrated but replication-backlog exists or "
+ "subs not created. Closing producers.", topic);
- } else {
- producers.forEach((__, producer) -> producer.topicMigrated(getMigratedClusterUrl()));
}
+ sendMigrationUrlFuture = CompletableFuture.completedFuture(null);
}
- producers.forEach((__, producer) -> futures.add(producer.disconnect()));
- disconnectProducersFuture = FutureUtil.waitForAll(futures);
+ disconnectProducersFuture = sendMigrationUrlFuture.thenCompose(v -> {
+ List<CompletableFuture<Void>> futures = new ArrayList<>();
+ producers.forEach((__, producer) -> futures.add(producer.disconnect()));
+ return FutureUtil.waitForAll(futures);
+ });
} else {
disconnectProducersFuture = CompletableFuture.completedFuture(null);
}
@@ -3264,39 +3269,39 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return CompletableFuture.completedFuture(null);
}
- Optional<ClusterUrl> clusterUrl = getMigratedClusterUrl();
-
- if (!clusterUrl.isPresent()) {
- return CompletableFuture.completedFuture(null);
- }
+ return getMigratedClusterUrlAsync().thenCompose(clusterUrl -> {
+ if (!clusterUrl.isPresent()) {
+ return CompletableFuture.<Void>completedFuture(null);
+ }
- if (isReplicated()) {
- if (isReplicationBacklogExist()) {
- if (!ledger.isMigrated()) {
- log.info("{} applying migration with replication backlog", topic);
- ledger.asyncMigrate();
- }
- if (log.isDebugEnabled()) {
- log.debug("{} has replication backlog and applied migration", topic);
+ if (isReplicated()) {
+ if (isReplicationBacklogExist()) {
+ if (!ledger.isMigrated()) {
+ log.info("{} applying migration with replication backlog", topic);
+ ledger.asyncMigrate();
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("{} has replication backlog and applied migration", topic);
+ }
+ return CompletableFuture.<Void>completedFuture(null);
}
- return CompletableFuture.completedFuture(null);
}
- }
- return initMigration().thenCompose(subCreated -> {
- migrationSubsCreated = true;
- CompletableFuture<?> migrated = !isMigrated() ? ledger.asyncMigrate()
- : CompletableFuture.completedFuture(null);
- return migrated.thenApply(__ -> {
- subscriptions.forEach((name, sub) -> {
- if (sub.isSubscriptionMigrated()) {
- sub.getConsumers().forEach(Consumer::checkAndApplyTopicMigration);
- }
- });
- return null;
- }).thenCompose(__ -> checkAndDisconnectReplicators())
- .thenCompose(__ -> checkAndUnsubscribeSubscriptions())
- .thenCompose(__ -> checkAndDisconnectProducers());
+ return initMigration().thenCompose(subCreated -> {
+ migrationSubsCreated = true;
+ CompletableFuture<?> migrated = !isMigrated() ? ledger.asyncMigrate()
+ : CompletableFuture.completedFuture(null);
+ return migrated.thenApply(__ -> {
+ subscriptions.forEach((name, sub) -> {
+ if (sub.isSubscriptionMigrated()) {
+ sub.getConsumers().forEach(consumer -> consumer.topicMigrated(clusterUrl));
+ }
+ });
+ return null;
+ }).thenCompose(__ -> checkAndDisconnectReplicators())
+ .thenCompose(__ -> checkAndUnsubscribeSubscriptions())
+ .thenCompose(__ -> checkAndDisconnectProducers());
+ });
});
}
(pulsar) 14/17: [fix][broker] Avoid blocking the dispatcher close path on delayed-delivery tracker close (#26053)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 8ca3fd0d890c7ece76cd4c5abe48d44fc7cf1352
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Thu Jun 18 04:23:09 2026 -0700
[fix][broker] Avoid blocking the dispatcher close path on delayed-delivery tracker close (#26053)
(cherry picked from commit 82074aa80c5ca8d849b640a368c254e38e5163e2)
---
.../broker/delayed/DelayedDeliveryTracker.java | 10 ++++++++
.../bucket/BucketDelayedDeliveryTracker.java | 27 ++++++++++++++--------
.../PersistentDispatcherMultipleConsumers.java | 8 ++++---
...rsistentDispatcherMultipleConsumersClassic.java | 8 ++++---
4 files changed, 38 insertions(+), 15 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/DelayedDeliveryTracker.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/DelayedDeliveryTracker.java
index 7c954879fe84..96e7151e7ea4 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/DelayedDeliveryTracker.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/DelayedDeliveryTracker.java
@@ -86,6 +86,16 @@ public interface DelayedDeliveryTracker extends AutoCloseable {
*/
void close();
+ /**
+ * Close the subscription tracker and release all resources, completing the returned future once the
+ * tracker has finished closing. Prefer this over {@link #close()} on asynchronous paths (e.g. inside a
+ * {@link CompletableFuture} continuation) so the caller is not blocked.
+ */
+ default CompletableFuture<Void> closeAsync() {
+ close();
+ return CompletableFuture.completedFuture(null);
+ }
+
DelayedDeliveryTracker DISABLE = new DelayedDeliveryTracker() {
@Override
public boolean addMessage(long ledgerId, long entryId, long deliveryAt) {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
index 9ed304c73312..77c4798232b5 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTracker.java
@@ -733,17 +733,26 @@ public class BucketDelayedDeliveryTracker extends AbstractDelayedDeliveryTracker
}
@Override
- public synchronized void close() {
- super.close();
- lastMutableBucket.close();
- sharedBucketPriorityQueue.close();
- try {
- List<CompletableFuture<Long>> completableFutures = immutableBuckets.asMapOfRanges().values().stream()
+ public void close() {
+ // Block for AutoCloseable / synchronous callers; asynchronous callers should use closeAsync().
+ closeAsync().join();
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ List<CompletableFuture<Long>> completableFutures;
+ synchronized (this) {
+ super.close();
+ lastMutableBucket.close();
+ sharedBucketPriorityQueue.close();
+ completableFutures = immutableBuckets.asMapOfRanges().values().stream()
.map(bucket -> bucket.getSnapshotCreateFuture().orElse(NULL_LONG_PROMISE)).toList();
- FutureUtil.waitForAll(completableFutures).get(AsyncOperationTimeoutSeconds, TimeUnit.SECONDS);
- } catch (Exception e) {
- log.warn("[{}] Failed wait to snapshot generate", dispatcher.getName(), e);
}
+ return FutureUtil.waitForAll(completableFutures)
+ .exceptionally(e -> {
+ log.warn("[{}] Failed wait to snapshot generate", dispatcher.getName(), e);
+ return null;
+ });
}
private CompletableFuture<Void> cleanImmutableBuckets() {
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index da6fc52b9163..e51f8a3acf8f 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -621,11 +621,13 @@ public class PersistentDispatcherMultipleConsumers extends AbstractPersistentDis
this.delayedDeliveryTracker = Optional.empty();
}
- delayedDeliveryTracker.ifPresent(DelayedDeliveryTracker::close);
+ CompletableFuture<Void> closeTrackerFuture = delayedDeliveryTracker
+ .map(DelayedDeliveryTracker::closeAsync)
+ .orElseGet(() -> CompletableFuture.completedFuture(null));
dispatchRateLimiter.ifPresent(DispatchRateLimiter::close);
- return disconnectConsumers
- ? disconnectAllConsumers(false, assignedBrokerLookupData) : CompletableFuture.completedFuture(null);
+ return closeTrackerFuture.thenCompose(__ -> disconnectConsumers
+ ? disconnectAllConsumers(false, assignedBrokerLookupData) : CompletableFuture.completedFuture(null));
}
@Override
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumersClassic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumersClassic.java
index e87b789d8b9f..0838b8deab7f 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumersClassic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumersClassic.java
@@ -534,11 +534,13 @@ public class PersistentDispatcherMultipleConsumersClassic extends AbstractPersis
this.delayedDeliveryTracker = Optional.empty();
}
- delayedDeliveryTracker.ifPresent(DelayedDeliveryTracker::close);
+ CompletableFuture<Void> closeTrackerFuture = delayedDeliveryTracker
+ .map(DelayedDeliveryTracker::closeAsync)
+ .orElseGet(() -> CompletableFuture.completedFuture(null));
dispatchRateLimiter.ifPresent(DispatchRateLimiter::close);
- return disconnectConsumers
- ? disconnectAllConsumers(false, assignedBrokerLookupData) : CompletableFuture.completedFuture(null);
+ return closeTrackerFuture.thenCompose(__ -> disconnectConsumers
+ ? disconnectAllConsumers(false, assignedBrokerLookupData) : CompletableFuture.completedFuture(null));
}
@Override
(pulsar) 02/17: [fix][broker] Fix forced topic/namespace deletion hanging or failing when compaction is in progress (#26016)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 55d299d0ba01634b4f1a39e4cd7f5c83516ce7a7
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sat Jun 13 08:53:57 2026 +0300
[fix][broker] Fix forced topic/namespace deletion hanging or failing when compaction is in progress (#26016)
(cherry picked from commit ec56aea8370e2dfdfda7aeeb1f5ee730d2d86866)
---
.../broker/service/persistent/PersistentTopic.java | 18 ++++---
.../apache/pulsar/client/impl/RawReaderImpl.java | 11 ++++
.../apache/pulsar/client/impl/RawReaderTest.java | 19 +++++++
.../apache/pulsar/compaction/CompactionTest.java | 58 ++++++++++++++++++++--
4 files changed, 95 insertions(+), 11 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index e08286024a44..38884047f2b9 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -1370,8 +1370,16 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
return;
}
}
- // Unsubscribe compaction cursor and delete compacted ledger.
- currentCompaction.thenCompose(__ -> {
+ // Unsubscribe compaction cursor and delete compacted ledger. Wait for any in-flight compaction to finish
+ // first, but don't let a compaction that completed exceptionally block the cursor deletion: the deletion
+ // would otherwise fail on every retry until the topic instance is reloaded (issue #24148). Note that a
+ // fenced topic makes the compactor's reader fail with an unrecoverable error, so a forced deletion
+ // terminates an in-flight compaction exceptionally rather than waiting for it to complete normally.
+ currentCompaction.exceptionally(compactionEx -> {
+ log.info("[{}][{}] Last compaction task failed, proceeding to delete the compaction cursor",
+ topic, subscriptionName, compactionEx);
+ return null;
+ }).thenCompose(__ -> {
asyncDeleteCursor(subscriptionName, unsubscribeFuture);
return unsubscribeFuture;
}).thenAccept(__ -> {
@@ -1389,11 +1397,7 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
disablingCompaction.compareAndSet(true, false);
}
}).exceptionally(ex -> {
- if (currentCompaction.isCompletedExceptionally()) {
- log.warn("[{}][{}] Last compaction task failed", topic, subscriptionName);
- } else {
- log.warn("[{}][{}] Failed to delete cursor task failed", topic, subscriptionName);
- }
+ log.warn("[{}][{}] Failed to delete the compaction cursor", topic, subscriptionName, ex);
// Reset the variable: disablingCompaction,
disablingCompaction.compareAndSet(true, false);
unsubscribeFuture.completeExceptionally(ex);
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
index c3f8254fb5e3..1bc88ed0532d 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/client/impl/RawReaderImpl.java
@@ -230,6 +230,17 @@ public class RawReaderImpl implements RawReader {
CompletableFuture<RawMessage> result = new CompletableFuture<>();
pendingRawReceives.add(result);
tryCompletePending();
+ // Once the consumer has reached a terminal state (for example it was closed after an
+ // unrecoverable error such as the topic or namespace being deleted), no further message
+ // will arrive and no close callback will run for receives enqueued from now on, so the
+ // future would never complete. Re-checking the state after enqueueing closes the race
+ // with a concurrent close() draining the queue, since close() drains only after moving to
+ // a terminal state. This matters for compaction: a never-completing read leaves the
+ // compaction future pending, which in turn blocks forced topic/namespace deletion.
+ State state = getState();
+ if (state == State.Closing || state == State.Closed || state == State.Failed) {
+ failPendingRawReceives();
+ }
return result;
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
index bb98ca1e4e77..9320e22e42e8 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/RawReaderTest.java
@@ -624,6 +624,25 @@ public class RawReaderTest extends MockedPulsarServiceBaseTest {
admin.topics().delete(topic, false);
}
+ @Test(timeOut = 30000)
+ public void testReadNextAsyncCompletesAfterConsumerClosed() throws Exception {
+ String topic = "persistent://my-property/my-ns/" + BrokerTestUtil.newUniqueName("reader");
+ admin.topics().createNonPartitionedTopic(topic);
+ RawReader reader = RawReader.create(pulsarClient, topic, subscription).get();
+
+ // Put the reader's underlying consumer into a terminal state. In production this happens when a
+ // compaction's RawReader hits an unrecoverable error (e.g. the topic/namespace is being deleted).
+ reader.closeAsync().get(5, TimeUnit.SECONDS);
+
+ // A read issued once the consumer has reached a terminal state must complete instead of hanging
+ // forever: a never-completing read keeps the compaction future pending, which blocks forced
+ // topic/namespace deletion (issue #24148).
+ CompletableFuture<RawMessage> readFuture = reader.readNextAsync();
+ Awaitility.await().atMost(10, TimeUnit.SECONDS)
+ .untilAsserted(() -> assertTrue(readFuture.isDone()));
+ assertTrue(readFuture.isCompletedExceptionally());
+ }
+
@Test(timeOut = 100000)
public void testPauseAndResume() throws Exception {
log.info("-- Starting testPauseAndResume test --");
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index 4adf3088b357..6ce5780f623e 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -136,6 +136,7 @@ public class CompactionTest extends MockedPulsarServiceBaseTest {
protected void doInitConf() throws Exception {
super.doInitConf();
conf.setDispatcherMaxReadBatchSize(1);
+ conf.setForceDeleteNamespaceAllowed(true);
}
@BeforeClass
@@ -2540,10 +2541,59 @@ public class CompactionTest extends MockedPulsarServiceBaseTest {
Thread.sleep(3000);
delayReadSignal.countDown();
- // Verify: topic deletion is successfully executed.
- Awaitility.await().atMost(15, TimeUnit.SECONDS).untilAsserted(() -> {
- assertTrue(deleteTopicFuture.isDone());
- });
+ // Verify: topic deletion is successfully executed. Asserting success (not just completion) covers the
+ // case where fencing the topic terminates the in-flight compaction exceptionally: the failed compaction
+ // must not fail the deletion (issue #24148).
+ deleteTopicFuture.get(15, TimeUnit.SECONDS);
+ }
+
+ @Test
+ public void testForcedDeleteSucceedsAfterFailedCompaction() throws Exception {
+ String topicName = newUniqueName("persistent://my-tenant/my-ns/delete-after-failed-compaction");
+ admin.topics().createNonPartitionedTopic(topicName);
+ try (Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic(topicName).create()) {
+ for (int i = 0; i < 10; i++) {
+ producer.newMessage().key("key" + (i % 2)).value("value-" + i).send();
+ }
+ }
+
+ // Fail the compaction after the phase-two seek
+ AbstractTwoPhaseCompactor.injectionPhaseTwoSeek =
+ (reader, id) -> CompletableFuture.failedFuture(new RuntimeException("injected compaction failure"));
+ PersistentTopic persistentTopic =
+ (PersistentTopic) pulsar.getBrokerService().getTopic(topicName, false).join().get();
+ persistentTopic.triggerCompaction();
+ Awaitility.await().untilAsserted(() ->
+ assertEquals(persistentTopic.compactionStatus().status, LongRunningProcessStatus.Status.ERROR));
+ AbstractTwoPhaseCompactor.injectionPhaseTwoSeek = RawReader::seekAsync;
+
+ // The failed compaction must not block the deletion of the compaction cursor
+ admin.topics().delete(topicName, true);
+ }
+
+ @Test
+ public void testForcedNamespaceDeleteWithInflightCompaction() throws Exception {
+ String namespace = "my-tenant/my-ns-inflight-compaction";
+ admin.namespaces().createNamespace(namespace, Set.of(configClusterName));
+ final String topicName = newUniqueName("persistent://" + namespace + "/inflight-compaction");
+ admin.topics().createNonPartitionedTopic(topicName);
+ try (Producer<String> producer = pulsarClient.newProducer(Schema.STRING).topic(topicName).create()) {
+ // dispatcherMaxReadBatchSize=1 makes the compactor read these one at a time, keeping the compaction
+ // in-flight for several seconds while the namespace is deleted
+ for (int i = 0; i < 2000; i++) {
+ producer.newMessage().key(String.valueOf(i)).value(String.valueOf(i)).send();
+ }
+ }
+ PersistentTopic persistentTopic =
+ (PersistentTopic) pulsar.getBrokerService().getTopic(topicName, false).join().get();
+ persistentTopic.triggerCompaction();
+ Awaitility.await().untilAsserted(() ->
+ assertEquals(persistentTopic.getSubscriptions().get(COMPACTION_SUBSCRIPTION).getConsumers().size(),
+ 1));
+
+ // Forced namespace deletion must succeed while the compaction is in-flight: fencing the topic terminates
+ // the compaction exceptionally, which must not fail the deletion of the compaction cursor (issue #24148)
+ deleteNamespaceWithRetry(namespace, true, admin);
}
@Test
(pulsar) 05/17: [fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl (#26031)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit c860f44350dfa4b475ac2d5f99d8989a3015409a
Author: Yunze Xu <xy...@163.com>
AuthorDate: Mon Jun 15 18:59:02 2026 +0800
[fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl (#26031)
(cherry picked from commit c3f8c05e4f27b194d889cbe3eefc4d83548628aa)
---
.../pulsar/broker/loadbalance/LoadManager.java | 52 ++++++++++------------
1 file changed, 24 insertions(+), 28 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadManager.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadManager.java
index 9399fcb3aad7..c9aa673d6b53 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadManager.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LoadManager.java
@@ -151,35 +151,31 @@ public interface LoadManager {
void initialize(PulsarService pulsar);
static LoadManager create(final PulsarService pulsar) {
- try {
- final ServiceConfiguration conf = pulsar.getConfiguration();
-
- String loadManagerClassName = conf.getLoadManagerClassName();
- if (StringUtils.isBlank(loadManagerClassName)) {
- loadManagerClassName = SimpleLoadManagerImpl.class.getName();
- }
-
- // Assume there is a constructor with one argument of PulsarService.
- final Object loadManagerInstance = Reflections.createInstance(loadManagerClassName,
- Thread.currentThread().getContextClassLoader());
- if (loadManagerInstance instanceof LoadManager casted) {
- casted.initialize(pulsar);
- return casted;
- } else if (loadManagerInstance instanceof ModularLoadManager modularLoadManager) {
- final LoadManager casted = new ModularLoadManagerWrapper(modularLoadManager);
- casted.initialize(pulsar);
- return casted;
- } else if (loadManagerInstance instanceof ExtensibleLoadManager) {
- final LoadManager casted =
- new ExtensibleLoadManagerWrapper((ExtensibleLoadManagerImpl) loadManagerInstance);
- casted.initialize(pulsar);
- return casted;
- }
- } catch (Exception e) {
- LOG.warn("Error when trying to create load manager: ", e);
+ final ServiceConfiguration conf = pulsar.getConfiguration();
+
+ String loadManagerClassName = conf.getLoadManagerClassName();
+ if (StringUtils.isBlank(loadManagerClassName)) {
+ loadManagerClassName = SimpleLoadManagerImpl.class.getName();
+ }
+
+ // Assume there is a constructor with one argument of PulsarService.
+ final Object loadManagerInstance = Reflections.createInstance(loadManagerClassName,
+ Thread.currentThread().getContextClassLoader());
+ if (loadManagerInstance instanceof LoadManager casted) {
+ casted.initialize(pulsar);
+ return casted;
+ } else if (loadManagerInstance instanceof ModularLoadManager modularLoadManager) {
+ final LoadManager casted = new ModularLoadManagerWrapper(modularLoadManager);
+ casted.initialize(pulsar);
+ return casted;
+ } else if (loadManagerInstance instanceof ExtensibleLoadManager) {
+ final LoadManager casted =
+ new ExtensibleLoadManagerWrapper((ExtensibleLoadManagerImpl) loadManagerInstance);
+ casted.initialize(pulsar);
+ return casted;
+ } else {
+ throw new IllegalArgumentException(loadManagerClassName + " is not a supported LoadManager");
}
- // If we failed to create a load manager, default to SimpleLoadManagerImpl.
- return new SimpleLoadManagerImpl(pulsar);
}
}
(pulsar) 04/17: [fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable (#26025)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit d42ac7a2d8d09ea4d47450293724700ffa0e2d63
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Jun 14 16:22:58 2026 +0300
[fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable (#26025)
(cherry picked from commit 24f2270ec21906608429d05f86ec7347e9bf0edf)
---
conf/broker.conf | 7 ++
conf/standalone.conf | 7 ++
.../apache/pulsar/broker/ServiceConfiguration.java | 13 ++
.../SystemTopicBasedTopicPoliciesService.java | 135 ++++++++++++++++++++-
.../SystemTopicBasedTopicPoliciesServiceTest.java | 121 ++++++++++++++++--
5 files changed, 271 insertions(+), 12 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 503a7a0e11f5..72a5f50876fa 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -769,6 +769,13 @@ systemTopicSchemaCompatibilityStrategy=ALWAYS_COMPATIBLE
# Please enable the system topic first.
topicLevelPoliciesEnabled=true
+# Amount of seconds to timeout initializing the topic policies cache of a namespace (reading the namespace's
+# __change_events system topic to the end). Topic loading waits for this initialization, so if the system-topic
+# reader gets stuck, this bounds the wait: the broker fails the initialization, closes the stuck reader and clears
+# the cached state so loading the namespace's topics can be retried with a fresh reader instead of hanging until the
+# broker is restarted. Set to 0 or a negative value to disable the timeout (not recommended).
+topicPoliciesCacheInitTimeoutSeconds=60
+
# If a topic remains fenced for this number of seconds, it will be closed forcefully.
# If it is set to 0 or a negative number, the fenced topic will not be closed.
topicFencingTimeoutSeconds=0
diff --git a/conf/standalone.conf b/conf/standalone.conf
index f9c7ccb658f3..59fcbdf2aabc 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -579,6 +579,13 @@ systemTopicSchemaCompatibilityStrategy=ALWAYS_COMPATIBLE
# Please enable the system topic first.
topicLevelPoliciesEnabled=true
+# Amount of seconds to timeout initializing the topic policies cache of a namespace (reading the namespace's
+# __change_events system topic to the end). Topic loading waits for this initialization, so if the system-topic
+# reader gets stuck, this bounds the wait: the broker fails the initialization, closes the stuck reader and clears
+# the cached state so loading the namespace's topics can be retried with a fresh reader instead of hanging until the
+# broker is restarted. Set to 0 or a negative value to disable the timeout (not recommended).
+topicPoliciesCacheInitTimeoutSeconds=60
+
# If a topic remains fenced for this number of seconds, it will be closed forcefully.
# If it is set to 0 or a negative number, the fenced topic will not be closed.
topicFencingTimeoutSeconds=0
diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 192f1b3ad4c6..4b4920bcc0be 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -552,6 +552,19 @@ public class ServiceConfiguration implements PulsarConfiguration {
)
private long topicLoadTimeoutSeconds = 60;
+ @FieldContext(
+ category = CATEGORY_SERVER,
+ dynamic = true,
+ doc = "Amount of seconds to timeout initializing the topic policies cache of a namespace (reading the "
+ + "namespace's __change_events system topic to the end). Topic loading waits for this "
+ + "initialization, so if the system-topic reader gets stuck (for example after __change_events is "
+ + "unloaded and the reconnected reader stops making progress), this bounds the wait: the broker "
+ + "fails the initialization, closes the stuck reader and clears the cached state so that loading "
+ + "the namespace's topics can be retried with a fresh reader instead of hanging until the broker "
+ + "is restarted. Set to 0 or a negative value to disable the timeout (not recommended)."
+ )
+ private long topicPoliciesCacheInitTimeoutSeconds = 60;
+
@FieldContext(
category = CATEGORY_SERVER,
doc = "Whether we should enable metadata operations batching"
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
index f8a5d5936610..7e1b453b35b8 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
@@ -24,6 +24,7 @@ import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
import com.github.benmanes.caffeine.cache.Caffeine;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.Sets;
+import io.opentelemetry.api.metrics.LongCounter;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@@ -34,7 +35,9 @@ import java.util.PriorityQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
@@ -78,10 +81,14 @@ import org.slf4j.LoggerFactory;
*/
public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesService {
+ public static final String TOPIC_POLICIES_CACHE_INIT_TIMEOUT_METRIC_NAME =
+ "pulsar.broker.topic.policies.cache.init.timeout.count";
+
private final PulsarService pulsarService;
private final HashSet<String> localCluster;
private final String clusterName;
private final AtomicBoolean closed = new AtomicBoolean(false);
+ private final LongCounter policyCacheInitTimeoutCounter;
private final ConcurrentInitializer<NamespaceEventsSystemTopicFactory>
namespaceEventsSystemTopicFactoryLazyInitializer = new LazyInitializer<>() {
@@ -125,6 +132,13 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
this.pulsarService = pulsarService;
this.clusterName = pulsarService.getConfiguration().getClusterName();
this.localCluster = Sets.newHashSet(clusterName);
+ this.policyCacheInitTimeoutCounter = pulsarService.getOpenTelemetry().getMeter()
+ .counterBuilder(TOPIC_POLICIES_CACHE_INIT_TIMEOUT_METRIC_NAME)
+ .setDescription("The number of times initializing a namespace's topic policies cache timed out "
+ + "because the __change_events system-topic reader was stuck. Each occurrence closes the "
+ + "stuck reader and clears the cached state so topic loading can be retried.")
+ .setUnit("{timeout}")
+ .build();
this.writerCaches = Caffeine.newBuilder()
.expireAfterAccess(5, TimeUnit.MINUTES)
.removalListener((namespaceName, writer, cause) -> {
@@ -583,6 +597,12 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
CompletableFuture<Void> existingFuture =
policyCacheInitMap.putIfAbsent(namespace, initNamespacePolicyFuture);
if (existingFuture == null) {
+ // Topic loading waits on this future, so a system-topic reader that gets stuck (e.g. after
+ // __change_events is unloaded and the reconnected reader stops making progress) would pin the
+ // policy cache for the whole namespace until the broker restarts (issue #25294). Bound the
+ // initialization so it fails fast and cleans up, letting topic loading retry with a fresh
+ // reader.
+ scheduleInitPolicesCacheTimeout(namespace, initNamespacePolicyFuture);
final CompletableFuture<SystemTopicClient.Reader<PulsarEvent>> readerCompletableFuture =
newReader(namespace);
readerCompletableFuture
@@ -597,14 +617,19 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
return null;
}).exceptionally(ex -> {
try {
+ // Identity-guarded cleanup: a concurrent timeout cleanup or a namespace unload
+ // may already have dropped this future and let a retry install a fresh
+ // future/reader, so clean up by namespace key here would clobber that newer
+ // attempt. Tear down state only while this future still owns the namespace.
if (readerCompletableFuture.isCompletedExceptionally()) {
log.error("[{}] Failed to create reader on __change_events topic",
namespace, ex);
initNamespacePolicyFuture.completeExceptionally(ex);
- cleanPoliciesCacheInitMap(namespace, true);
+ cleanupFailedPolicyCacheInit(namespace, initNamespacePolicyFuture, true);
} else {
initNamespacePolicyFuture.completeExceptionally(ex);
- cleanPoliciesCacheInitMap(namespace, isAlreadyClosedException(ex));
+ cleanupFailedPolicyCacheInit(namespace, initNamespacePolicyFuture,
+ isAlreadyClosedException(ex));
}
} catch (Throwable cleanupEx) {
// Adding this catch to avoid break callback chain
@@ -685,6 +710,86 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
});
}
+ /**
+ * Bound the topic-policies cache initialization for {@code namespace} so a stuck {@code __change_events} reader
+ * cannot pin the cache (and therefore topic loading) for the whole namespace indefinitely (issue #25294). If the
+ * timeout wins the race to complete {@code initNamespacePolicyFuture}, the cached state is cleared and the stuck
+ * reader is closed so a subsequent load retries from scratch rather than waiting until the broker restarts.
+ */
+ private void scheduleInitPolicesCacheTimeout(@NonNull NamespaceName namespace,
+ @NonNull CompletableFuture<Void> initNamespacePolicyFuture) {
+ long timeoutSeconds = pulsarService.getConfiguration().getTopicPoliciesCacheInitTimeoutSeconds();
+ if (timeoutSeconds <= 0) {
+ return;
+ }
+ final ScheduledFuture<?> timeoutTask = pulsarService.getExecutor().schedule(() -> {
+ TimeoutException timeoutException = new TimeoutException(String.format(
+ "Timed out after %d seconds initializing the topic policies cache for namespace %s; the "
+ + "__change_events reader did not reach the end of the topic", timeoutSeconds, namespace));
+ if (initNamespacePolicyFuture.completeExceptionally(timeoutException)) {
+ policyCacheInitTimeoutCounter.add(1);
+ log.error("[{}] Timed out initializing the topic policies cache after {} seconds; closing the stuck "
+ + "__change_events reader so the namespace can be loaded again", namespace, timeoutSeconds);
+ try {
+ cleanupFailedPolicyCacheInit(namespace, initNamespacePolicyFuture, true);
+ } catch (Throwable cleanupEx) {
+ log.error("[{}] Failed to clean up the topic policies cache after init timeout", namespace,
+ cleanupEx);
+ }
+ }
+ }, timeoutSeconds, TimeUnit.SECONDS);
+ // Cancel the timeout once initialization finishes (successfully or not) so we don't leak scheduled tasks.
+ initNamespacePolicyFuture.whenComplete((__, ex) -> timeoutTask.cancel(false));
+ }
+
+ /**
+ * Identity-guarded cleanup for an initialization that failed (it timed out, the {@code __change_events} reader
+ * could not be created, or reading the topic threw). Unlike {@link #cleanPoliciesCacheInitMap}, which
+ * removes/closes by namespace key unconditionally, this only tears down state that still belongs to
+ * {@code initFuture}. By the time the failure is observed, a concurrent retry — or a namespace-bundle unload that
+ * left the init future orphaned — may already own the namespace with a fresh future and reader; removing by key
+ * would drop that newer future and close its reader, pinning the namespace again. Guarding on identity ensures a
+ * late failure never clobbers a newer initialization.
+ *
+ * @param closeReader when {@code true}, also clears the cached policies and closes the reader that belongs to this
+ * initialization; when {@code false}, only the init future is dropped, leaving the reader cached
+ * for the retry to reuse (mirrors the transient read-error path of
+ * {@link #cleanPoliciesCacheInitMap}).
+ */
+ @VisibleForTesting
+ void cleanupFailedPolicyCacheInit(@NonNull NamespaceName namespace,
+ @NonNull CompletableFuture<Void> initFuture, boolean closeReader) {
+ // Capture the reader before dropping the init future so we only close the reader that belongs to this
+ // initialization, never one a concurrent retry creates immediately afterwards.
+ CompletableFuture<SystemTopicClient.Reader<PulsarEvent>> readerFuture =
+ closeReader ? readerCaches.get(namespace) : null;
+ if (!policyCacheInitMap.remove(namespace, initFuture)) {
+ // Superseded by a retry or an unload; that owner is responsible for its own reader/state.
+ return;
+ }
+ // Complete the dropped future (a no-op if the caller already completed it) outside any map remapping function,
+ // so awaiting topic loads fail fast and retry instead of hanging until the broker restarts (issue #25294).
+ failPendingPolicyCacheInit(namespace, initFuture);
+ if (!closeReader) {
+ return;
+ }
+ policiesCache.entrySet().removeIf(entry -> Objects.equals(entry.getKey().getNamespaceObject(), namespace));
+ globalPoliciesCache.entrySet()
+ .removeIf(entry -> Objects.equals(entry.getKey().getNamespaceObject(), namespace));
+ TopicPolicyMessageHandlerTracker tracker = topicPolicyMessageHandlerTrackers.remove(namespace);
+ if (tracker != null) {
+ tracker.close();
+ }
+ if (readerFuture != null && readerCaches.remove(namespace, readerFuture)
+ && !readerFuture.isCompletedExceptionally()) {
+ readerFuture.thenCompose(SystemTopicClient.Reader::closeAsync)
+ .exceptionally(ex -> {
+ log.warn("[{}] Close change_event reader fail.", namespace, ex);
+ return null;
+ });
+ }
+ }
+
private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, CompletableFuture<Void> future) {
if (closed.get()) {
future.completeExceptionally(new BrokerServiceException(getClass().getName() + " is closed."));
@@ -742,7 +847,7 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
@VisibleForTesting
void cleanPoliciesCacheInitMap(@NonNull NamespaceName namespace, boolean closeReader) {
if (!closeReader) {
- policyCacheInitMap.remove(namespace);
+ failPendingPolicyCacheInit(namespace, policyCacheInitMap.remove(namespace));
return;
}
@@ -753,12 +858,18 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
}
CompletableFuture<SystemTopicClient.Reader<PulsarEvent>> readerFuture = readerCaches.remove(namespace);
+ MutableObject<CompletableFuture<Void>> removedInitFuture = new MutableObject<>();
policyCacheInitMap.compute(namespace, (k, v) -> {
+ removedInitFuture.setValue(v);
policiesCache.entrySet().removeIf(entry -> Objects.equals(entry.getKey().getNamespaceObject(), namespace));
globalPoliciesCache.entrySet()
.removeIf(entry -> Objects.equals(entry.getKey().getNamespaceObject(), namespace));
return null;
});
+ // Complete the removed init future outside the compute() remapping function: completing it can run the
+ // awaiting topic-load callbacks synchronously, and doing that while holding the ConcurrentHashMap bin lock
+ // risks a recursive map update / deadlock (see #24977).
+ failPendingPolicyCacheInit(namespace, removedInitFuture.getValue());
if (readerFuture != null && !readerFuture.isCompletedExceptionally()) {
readerFuture
.thenCompose(SystemTopicClient.Reader::closeAsync)
@@ -769,6 +880,20 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
}
}
+ /**
+ * Complete an init future that is being dropped from {@link #policyCacheInitMap} but never completed, so the
+ * topic loads awaiting it fail fast and retry instead of hanging until the broker restarts (issue #25294).
+ * No-op if the future was already completed by its own initialization chain.
+ */
+ private void failPendingPolicyCacheInit(@NonNull NamespaceName namespace,
+ @Nullable CompletableFuture<Void> initFuture) {
+ if (initFuture != null && !initFuture.isDone()) {
+ initFuture.completeExceptionally(new BrokerServiceException(
+ "Topic policies cache initialization for namespace " + namespace
+ + " was aborted because the cached state was cleared"));
+ }
+ }
+
private void cleanWriterCache(@NonNull NamespaceName namespace) {
writerCaches.synchronous().invalidate(namespace);
}
@@ -1033,6 +1158,10 @@ public class SystemTopicBasedTopicPoliciesService implements TopicPoliciesServic
}
});
readerCaches.clear();
+ // Release any topic loads still waiting on an in-progress policy cache initialization so they fail
+ // fast instead of hanging until the awaited future is completed indirectly (issue #25294).
+ policyCacheInitMap.forEach(this::failPendingPolicyCacheInit);
+ policyCacheInitMap.clear();
}
}
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
index 8d3b16723ed4..1b5182a45452 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java
@@ -28,6 +28,7 @@ import static org.testng.AssertJUnit.assertEquals;
import static org.testng.AssertJUnit.assertFalse;
import static org.testng.AssertJUnit.assertNotNull;
import static org.testng.AssertJUnit.assertNull;
+import static org.testng.AssertJUnit.assertSame;
import static org.testng.AssertJUnit.assertTrue;
import java.time.Duration;
import java.util.HashSet;
@@ -41,12 +42,15 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
import lombok.Cleanup;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
import org.apache.pulsar.broker.systopic.SystemTopicClient;
import org.apache.pulsar.client.admin.PulsarAdminException;
+import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.common.events.PulsarEvent;
import org.apache.pulsar.common.naming.NamespaceName;
@@ -554,17 +558,18 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
Assert.assertNull(readerCompletableFuture1);
});
-
- // make sure not do cleanPoliciesCacheInitMap() twice
- // totally trigger prepareInitPoliciesCacheAsync() twice, so the time of cleanPoliciesCacheInitMap() is 2.
- // in previous code, the time would be 3
+ // Cleanup must run exactly once per trigger and not repeat recursively (in older code it ran 3 times).
+ // Two failures are triggered here: the reader.close() above drives readMorePoliciesAsync into
+ // cleanPoliciesCacheInitMap (1x), and the second prepareInitPoliciesCacheAsync fails in initPolicesCache and
+ // is torn down by the identity-guarded cleanupFailedPolicyCacheInit (1x).
boolean logFound = testLogAppender.getEvents().stream().anyMatch(logEvent ->
logEvent.getMessage().toString().contains("Failed to create reader on __change_events topic"));
assertFalse(logFound);
boolean logFound2 = testLogAppender.getEvents().stream().anyMatch(logEvent ->
logEvent.getMessage().toString().contains("Failed to check the move events for the system topic"));
assertTrue(logFound2);
- verify(spyService, times(2)).cleanPoliciesCacheInitMap(any(), anyBoolean());
+ verify(spyService, times(1)).cleanPoliciesCacheInitMap(any(), anyBoolean());
+ verify(spyService, times(1)).cleanupFailedPolicyCacheInit(any(), any(), anyBoolean());
// make sure not occur Recursive update
boolean logFound3 = testLogAppender.getEvents().stream().anyMatch(logEvent ->
@@ -623,9 +628,9 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
Assert.assertNull(readerCompletableFuture1);
});
-
- // make sure not do cleanPoliciesCacheInitMap() twice
- // totally trigger prepareInitPoliciesCacheAsync() once, so the time of cleanPoliciesCacheInitMap() is 1.
+ // Reader creation fails, so the single cleanup runs once via the identity-guarded
+ // cleanupFailedPolicyCacheInit (the reader-creation-failure branch no longer goes through
+ // cleanPoliciesCacheInitMap), and must not run more than once.
boolean logFound = testLogAppender.getEvents().stream().anyMatch(logEvent ->
logEvent.getMessage().toString().contains("Failed to create reader on __change_events topic"));
assertTrue(logFound);
@@ -633,6 +638,104 @@ public class SystemTopicBasedTopicPoliciesServiceTest extends MockedPulsarServic
logEvent.getMessage().toString().contains("Failed to check the move events for the system topic")
|| logEvent.getMessage().toString().contains("Failed to read event from the system topic"));
assertFalse(logFound2);
- verify(spyService, times(1)).cleanPoliciesCacheInitMap(any(), anyBoolean());
+ verify(spyService, times(1)).cleanupFailedPolicyCacheInit(any(), any(), anyBoolean());
+ verify(spyService, times(0)).cleanPoliciesCacheInitMap(any(), anyBoolean());
+ }
+
+ @Test(timeOut = 60_000)
+ public void testPrepareInitPoliciesCacheAsyncTimesOutWhenReaderStuck() throws Exception {
+ // Bound the policy-cache initialization to a short timeout for the test.
+ pulsar.getConfiguration().setTopicPoliciesCacheInitTimeoutSeconds(3);
+
+ pulsar.getTopicPoliciesService().close();
+ SystemTopicBasedTopicPoliciesService spyService =
+ Mockito.spy(new SystemTopicBasedTopicPoliciesService(pulsar));
+ FieldUtils.writeField(pulsar, "topicPoliciesService", spyService, true);
+
+ admin.namespaces().createNamespace(NAMESPACE5);
+ final NamespaceName namespace = NamespaceName.get(NAMESPACE5);
+
+ // Create a real __change_events reader, then spy it so that it reports more events but never delivers one —
+ // i.e. a reader that reconnected but is stuck (issue #25294). initPolicesCache would otherwise never complete.
+ SystemTopicClient.Reader<PulsarEvent> stuckReader = Mockito.spy(spyService.createSystemTopicClient(namespace)
+ .get(30, TimeUnit.SECONDS));
+ Mockito.doReturn(CompletableFuture.completedFuture(true)).when(stuckReader).hasMoreEventsAsync();
+ Mockito.doReturn(new CompletableFuture<Message<PulsarEvent>>()).when(stuckReader).readNextAsync();
+ Mockito.doReturn(CompletableFuture.completedFuture(stuckReader))
+ .when(spyService).createSystemTopicClient(namespace);
+
+ // Without the timeout the returned future never completes and topic loading for the namespace hangs forever.
+ CompletableFuture<Boolean> prepareFuture = spyService.prepareInitPoliciesCacheAsync(namespace);
+ try {
+ prepareFuture.get(15, TimeUnit.SECONDS);
+ Assert.fail("Expected the topic policies cache initialization to time out");
+ } catch (ExecutionException e) {
+ assertTrue("Expected a TimeoutException cause but got " + e.getCause(),
+ e.getCause() instanceof TimeoutException);
+ }
+
+ // The poisoned cache entry must be cleared and the stuck reader closed so a subsequent load can retry with a
+ // fresh reader instead of being pinned until the broker restarts.
+ Awaitility.await().untilAsserted(() -> assertNull(spyService.getPoliciesCacheInit(namespace)));
+ Mockito.verify(stuckReader, Mockito.atLeastOnce()).closeAsync();
+ }
+
+ @Test
+ public void testCleanPoliciesCacheInitMapCompletesPendingInitFuture() {
+ SystemTopicBasedTopicPoliciesService service =
+ (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+ final NamespaceName namespace = NamespaceName.get(NAMESPACE1);
+
+ // Dropping the cached init future (e.g. on a namespace-bundle unload) must complete it so the topic loads
+ // awaiting it fail fast and retry, instead of hanging until the broker restarts (issue #25294).
+ CompletableFuture<Void> pendingWithReaderClose = new CompletableFuture<>();
+ service.policyCacheInitMap.put(namespace, pendingWithReaderClose);
+ service.cleanPoliciesCacheInitMap(namespace, true);
+ assertTrue(pendingWithReaderClose.isCompletedExceptionally());
+ assertNull(service.getPoliciesCacheInit(namespace));
+
+ CompletableFuture<Void> pendingWithoutReaderClose = new CompletableFuture<>();
+ service.policyCacheInitMap.put(namespace, pendingWithoutReaderClose);
+ service.cleanPoliciesCacheInitMap(namespace, false);
+ assertTrue(pendingWithoutReaderClose.isCompletedExceptionally());
+ assertNull(service.getPoliciesCacheInit(namespace));
+
+ // An already-completed init future must not be overwritten/disturbed.
+ CompletableFuture<Void> alreadyDone = CompletableFuture.completedFuture(null);
+ service.policyCacheInitMap.put(namespace, alreadyDone);
+ service.cleanPoliciesCacheInitMap(namespace, true);
+ assertFalse(alreadyDone.isCompletedExceptionally());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testCleanupFailedPolicyCacheInitIsIdentityGuarded() {
+ SystemTopicBasedTopicPoliciesService service =
+ (SystemTopicBasedTopicPoliciesService) pulsar.getTopicPoliciesService();
+ final NamespaceName namespace = NamespaceName.get(NAMESPACE1);
+
+ // A newer init attempt (B) already owns the namespace with a fresh future and reader.
+ CompletableFuture<Void> newerInitFuture = new CompletableFuture<>();
+ SystemTopicClient.Reader<PulsarEvent> newerReader = Mockito.mock(SystemTopicClient.Reader.class);
+ Mockito.doReturn(CompletableFuture.completedFuture(null)).when(newerReader).closeAsync();
+ service.policyCacheInitMap.put(namespace, newerInitFuture);
+ service.getReaderCaches().put(namespace, CompletableFuture.completedFuture(newerReader));
+
+ // A stale init attempt (A) whose reader was already torn down (e.g. by the timeout cleanup) fires its failure
+ // callback late. Cleaning it up by identity must be a no-op: it must not clobber B's future or close B's reader
+ // (issue #25294 follow-up).
+ CompletableFuture<Void> staleInitFuture = new CompletableFuture<>();
+ service.cleanupFailedPolicyCacheInit(namespace, staleInitFuture, true);
+ assertSame(newerInitFuture, service.getPoliciesCacheInit(namespace));
+ assertFalse(newerInitFuture.isDone());
+ assertNotNull(service.getReaderCaches().get(namespace));
+ Mockito.verify(newerReader, Mockito.never()).closeAsync();
+
+ // Cleaning up the owning attempt does drop its future and close its reader.
+ service.cleanupFailedPolicyCacheInit(namespace, newerInitFuture, true);
+ assertNull(service.getPoliciesCacheInit(namespace));
+ assertTrue(newerInitFuture.isCompletedExceptionally());
+ assertNull(service.getReaderCaches().get(namespace));
+ Mockito.verify(newerReader, Mockito.times(1)).closeAsync();
}
}
(pulsar) 15/17: [fix][broker] Avoid blocking the bundle-throughput lookup on per-bundle metadata reads (#26054)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9f4d11e63d836b92ee013973d31c7e85c877d19c
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Thu Jun 18 04:25:00 2026 -0700
[fix][broker] Avoid blocking the bundle-throughput lookup on per-bundle metadata reads (#26054)
(cherry picked from commit 561f373c1442a81364154aa9880fcb7ada9b89ef)
---
.../broker/loadbalance/ModularLoadManager.java | 14 ++++
.../loadbalance/impl/ModularLoadManagerImpl.java | 89 ++++++++++++----------
.../common/naming/NamespaceBundleFactory.java | 31 +++++---
.../impl/ModularLoadManagerImplTest.java | 2 +-
.../broker/namespace/NamespaceServiceTest.java | 2 +-
5 files changed, 84 insertions(+), 54 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManager.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManager.java
index d608bd6784f2..93059d722e73 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManager.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManager.java
@@ -138,8 +138,22 @@ public interface ModularLoadManager {
*
* @param bundle
* @return bundle data
+ * @deprecated use {@link #getBundleDataOrDefaultAsync(String)} instead; this method blocks on
+ * metadata-store reads and must not be called from async or event-loop threads.
*/
+ @Deprecated
BundleData getBundleDataOrDefault(String bundle);
+ /**
+ * Asynchronously fetch bundle's load report data.
+ *
+ * @param bundle
+ * @return future of the bundle data
+ */
+ @SuppressWarnings("deprecation")
+ default CompletableFuture<BundleData> getBundleDataOrDefaultAsync(String bundle) {
+ return CompletableFuture.completedFuture(getBundleDataOrDefault(bundle));
+ }
+
String setNamespaceBundleAffinity(String bundle, String broker);
}
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
index a9d7ddd78e07..214d12e7ad2c 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java
@@ -381,46 +381,55 @@ public class ModularLoadManagerImpl implements ModularLoadManager {
// Attempt to local the data for the given bundle in metadata store
// If it cannot be found, return the default bundle data.
+ /**
+ * @deprecated use {@link #getBundleDataOrDefaultAsync(String)} instead.
+ */
+ @Deprecated
@Override
public BundleData getBundleDataOrDefault(final String bundle) {
- BundleData bundleData = null;
- try {
- Optional<BundleData> optBundleData =
- pulsarResources.getLoadBalanceResources().getBundleDataResources().getBundleData(bundle).join();
- if (optBundleData.isPresent()) {
- return optBundleData.get();
- }
+ return getBundleDataOrDefaultAsync(bundle).join();
+ }
- Optional<ResourceQuota> optQuota = pulsarResources.getLoadBalanceResources().getQuotaResources()
- .getQuota(bundle).join();
- if (optQuota.isPresent()) {
- ResourceQuota quota = optQuota.get();
- bundleData = new BundleData(NUM_SHORT_SAMPLES, NUM_LONG_SAMPLES);
- // Initialize from existing resource quotas if new API ZNodes do not exist.
- final TimeAverageMessageData shortTermData = bundleData.getShortTermData();
- final TimeAverageMessageData longTermData = bundleData.getLongTermData();
-
- shortTermData.setMsgRateIn(quota.getMsgRateIn());
- shortTermData.setMsgRateOut(quota.getMsgRateOut());
- shortTermData.setMsgThroughputIn(quota.getBandwidthIn());
- shortTermData.setMsgThroughputOut(quota.getBandwidthOut());
-
- longTermData.setMsgRateIn(quota.getMsgRateIn());
- longTermData.setMsgRateOut(quota.getMsgRateOut());
- longTermData.setMsgThroughputIn(quota.getBandwidthIn());
- longTermData.setMsgThroughputOut(quota.getBandwidthOut());
-
- // Assume ample history.
- shortTermData.setNumSamples(NUM_SHORT_SAMPLES);
- longTermData.setNumSamples(NUM_LONG_SAMPLES);
- }
- } catch (Exception e) {
- log.warn("Error when trying to find bundle {} on metadata store: {}", bundle, e);
- }
- if (bundleData == null) {
- bundleData = new BundleData(NUM_SHORT_SAMPLES, NUM_LONG_SAMPLES, defaultStats);
- }
- return bundleData;
+ @Override
+ public CompletableFuture<BundleData> getBundleDataOrDefaultAsync(final String bundle) {
+ return pulsarResources.getLoadBalanceResources().getBundleDataResources().getBundleData(bundle)
+ .thenCompose(optBundleData -> {
+ if (optBundleData.isPresent()) {
+ return CompletableFuture.completedFuture(optBundleData.get());
+ }
+ return pulsarResources.getLoadBalanceResources().getQuotaResources().getQuota(bundle)
+ .thenApply(optQuota -> {
+ if (optQuota.isEmpty()) {
+ return null;
+ }
+ ResourceQuota quota = optQuota.get();
+ BundleData bundleData = new BundleData(NUM_SHORT_SAMPLES, NUM_LONG_SAMPLES);
+ // Initialize from existing resource quotas if new API ZNodes do not exist.
+ final TimeAverageMessageData shortTermData = bundleData.getShortTermData();
+ final TimeAverageMessageData longTermData = bundleData.getLongTermData();
+
+ shortTermData.setMsgRateIn(quota.getMsgRateIn());
+ shortTermData.setMsgRateOut(quota.getMsgRateOut());
+ shortTermData.setMsgThroughputIn(quota.getBandwidthIn());
+ shortTermData.setMsgThroughputOut(quota.getBandwidthOut());
+
+ longTermData.setMsgRateIn(quota.getMsgRateIn());
+ longTermData.setMsgRateOut(quota.getMsgRateOut());
+ longTermData.setMsgThroughputIn(quota.getBandwidthIn());
+ longTermData.setMsgThroughputOut(quota.getBandwidthOut());
+
+ // Assume ample history.
+ shortTermData.setNumSamples(NUM_SHORT_SAMPLES);
+ longTermData.setNumSamples(NUM_LONG_SAMPLES);
+ return bundleData;
+ });
+ })
+ .exceptionally(e -> {
+ log.warn("Error when trying to find bundle {} on metadata store: {}", bundle, e);
+ return null;
+ })
+ .thenApply(bundleData -> bundleData != null ? bundleData
+ : new BundleData(NUM_SHORT_SAMPLES, NUM_LONG_SAMPLES, defaultStats));
}
// Use the Pulsar client to acquire the namespace bundle stats.
@@ -561,7 +570,7 @@ public class ModularLoadManagerImpl implements ModularLoadManager {
} else {
// Otherwise, attempt to find the bundle data on metadata store.
// If it cannot be found, use the latest stats as the first sample.
- BundleData currentBundleData = getBundleDataOrDefault(bundle);
+ BundleData currentBundleData = getBundleDataOrDefaultAsync(bundle).join();
currentBundleData.update(stats);
bundleData.put(bundle, currentBundleData);
}
@@ -879,7 +888,7 @@ public class ModularLoadManagerImpl implements ModularLoadManager {
private void preallocateBundle(String bundle, String broker) {
final BundleData data = loadData.getBundleData().computeIfAbsent(bundle,
- key -> getBundleDataOrDefault(bundle));
+ key -> getBundleDataOrDefaultAsync(bundle).join());
loadData.getBrokerData().get(broker).getPreallocatedBundleData().put(bundle, data);
preallocatedBundleToBroker.put(bundle, broker);
@@ -893,7 +902,7 @@ public class ModularLoadManagerImpl implements ModularLoadManager {
synchronized (brokerCandidateCache) {
final String bundle = serviceUnit.toString();
final BundleData data = loadData.getBundleData().computeIfAbsent(bundle,
- key -> getBundleDataOrDefault(bundle));
+ key -> getBundleDataOrDefaultAsync(bundle).join());
brokerCandidateCache.clear();
LoadManagerShared.applyNamespacePolicies(serviceUnit, policies, brokerCandidateCache,
getAvailableBrokers(),
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java b/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java
index 69f5208ce671..846f64c32c09 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/NamespaceBundleFactory.java
@@ -56,6 +56,7 @@ import org.apache.pulsar.common.policies.data.LocalPolicies;
import org.apache.pulsar.common.policies.data.Policies;
import org.apache.pulsar.common.stats.CacheMetricsCollector;
import org.apache.pulsar.common.util.Backoff;
+import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.metadata.api.Notification;
import org.apache.pulsar.policies.data.loadbalancer.BundleData;
import org.slf4j.Logger;
@@ -244,19 +245,25 @@ public class NamespaceBundleFactory {
public CompletableFuture<NamespaceBundle> getBundleWithHighestThroughputAsync(NamespaceName nsName) {
LoadManager loadManager = pulsar.getLoadManager().get();
if (loadManager instanceof ModularLoadManagerWrapper) {
- return getBundlesAsync(nsName).thenApply(bundles -> {
- double maxMsgThroughput = -1;
- NamespaceBundle bundleWithHighestThroughput = null;
- for (NamespaceBundle bundle : bundles.getBundles()) {
- BundleData bundleData = ((ModularLoadManagerWrapper) loadManager).getLoadManager()
- .getBundleDataOrDefault(bundle.toString());
- if (bundleData.getTopics() > 0
- && bundleData.getLongTermData().totalMsgThroughput() > maxMsgThroughput) {
- maxMsgThroughput = bundleData.getLongTermData().totalMsgThroughput();
- bundleWithHighestThroughput = bundle;
+ return getBundlesAsync(nsName).thenCompose(bundles -> {
+ List<NamespaceBundle> bundleList = bundles.getBundles();
+ List<CompletableFuture<BundleData>> bundleDataFutures = bundleList.stream()
+ .map(bundle -> ((ModularLoadManagerWrapper) loadManager).getLoadManager()
+ .getBundleDataOrDefaultAsync(bundle.toString()))
+ .toList();
+ return FutureUtil.waitForAll(bundleDataFutures).thenApply(__ -> {
+ double maxMsgThroughput = -1;
+ NamespaceBundle bundleWithHighestThroughput = null;
+ for (int i = 0; i < bundleList.size(); i++) {
+ BundleData bundleData = bundleDataFutures.get(i).join();
+ if (bundleData.getTopics() > 0
+ && bundleData.getLongTermData().totalMsgThroughput() > maxMsgThroughput) {
+ maxMsgThroughput = bundleData.getLongTermData().totalMsgThroughput();
+ bundleWithHighestThroughput = bundleList.get(i);
+ }
}
- }
- return bundleWithHighestThroughput;
+ return bundleWithHighestThroughput;
+ });
});
}
return getBundleWithHighestTopicsAsync(nsName);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImplTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImplTest.java
index f6061204b62a..60486c96a184 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImplTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImplTest.java
@@ -1026,7 +1026,7 @@ public class ModularLoadManagerImplTest {
// get the bundleData of the first bundle range.
// The default value of the bundleData be the same as resourceQuota because the resourceQuota is present.
- BundleData defaultBundleData = lm.getBundleDataOrDefault(namespaceBundle.toString());
+ BundleData defaultBundleData = lm.getBundleDataOrDefaultAsync(namespaceBundle.toString()).join();
TimeAverageMessageData shortTermData = defaultBundleData.getShortTermData();
TimeAverageMessageData longTermData = defaultBundleData.getLongTermData();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/NamespaceServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/NamespaceServiceTest.java
index 6fcebff2b81a..7193810bec47 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/NamespaceServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/NamespaceServiceTest.java
@@ -688,7 +688,7 @@ public class NamespaceServiceTest extends BrokerTestBase {
LoadManager loadManager = pulsar.getLoadManager().get();
Awaitility.await().untilAsserted(() -> {
BundleData targetBundleData = ((ModularLoadManagerWrapper) loadManager).getLoadManager()
- .getBundleDataOrDefault(namespace + "/" + bundle);
+ .getBundleDataOrDefaultAsync(namespace + "/" + bundle).join();
assertEquals(targetBundleData.getTopics(), 10);
});
(pulsar) 01/17: [fix][broker] Prevent subscribe rate limit from stalling compaction and blocking forced deletion (#26015)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9124855f8436b1dc67070c59506431750bbace8e
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sat Jun 13 04:55:20 2026 +0300
[fix][broker] Prevent subscribe rate limit from stalling compaction and blocking forced deletion (#26015)
(cherry picked from commit a1b5a0dbd8d67f3fcb5db4db53f73e7acd8c825c)
---
.../broker/service/persistent/PersistentTopic.java | 11 ++++++--
.../apache/pulsar/compaction/CompactionTest.java | 29 ++++++++++++++++++++++
2 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index df01fcfce891..e08286024a44 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -987,8 +987,15 @@ public class PersistentTopic extends AbstractTopic implements Topic, AddEntryCal
new NamingException("Subscription with reserved subscription name attempted"));
}
- if (cnx.clientAddress() != null && cnx.clientAddress().toString().contains(":")
- && subscribeRateLimiter.isPresent()) {
+ // The subscribe rate limit must not apply to the broker-internal compaction subscription: the
+ // compactor's reader re-subscribes after the phase-two seek, and throttling that re-subscribe stalls
+ // the compaction, which in turn blocks forced topic/namespace deletion waiting on the in-flight
+ // compaction. System topics are exempt as well, consistent with the publish/dispatch rate limiters,
+ // since throttling broker-internal readers (e.g. on __change_events) can stall topic policy updates.
+ if (subscribeRateLimiter.isPresent()
+ && !isSystemTopic()
+ && !isCompactionSubscription(subscriptionName)
+ && cnx.clientAddress() != null && cnx.clientAddress().toString().contains(":")) {
SubscribeRateLimiter.ConsumerIdentifier consumer = new SubscribeRateLimiter.ConsumerIdentifier(
cnx.clientAddress().toString().split(":")[0], consumerName, consumerId);
if (!subscribeRateLimiter.get().subscribeAvailable(consumer)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
index 1d4856fe0536..4adf3088b357 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactionTest.java
@@ -110,6 +110,7 @@ import org.apache.pulsar.common.naming.TopicName;
import org.apache.pulsar.common.policies.data.ClusterData;
import org.apache.pulsar.common.policies.data.PersistentTopicInternalStats;
import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.SubscribeRate;
import org.apache.pulsar.common.policies.data.TenantInfoImpl;
import org.apache.pulsar.common.protocol.Markers;
import org.apache.pulsar.common.util.FutureUtil;
@@ -196,6 +197,34 @@ public class CompactionTest extends MockedPulsarServiceBaseTest {
return compactor;
}
+ @Test
+ public void testCompactionNotBlockedBySubscribeRateLimit() throws Exception {
+ String namespace = "my-tenant/my-ns";
+ String topic = "persistent://" + namespace + "/compaction-with-subscribe-rate-limit";
+
+ try (Producer<byte[]> producer = pulsarClient.newProducer().topic(topic).enableBatching(false).create()) {
+ for (int i = 0; i < 10; i++) {
+ producer.newMessage().key("key" + (i % 2)).value(("my-message-" + i).getBytes()).send();
+ }
+ }
+
+ // Allow a single subscribe per consumer within a long period. The compactor's reader consumes the only
+ // token when it first subscribes, so the re-subscribe triggered by the phase-two seek would be throttled
+ // if the limit applied to the compaction subscription, stalling the compaction.
+ admin.namespaces().setSubscribeRate(namespace, new SubscribeRate(1, 3600));
+ Awaitility.await().untilAsserted(() -> {
+ PersistentTopic persistentTopic =
+ (PersistentTopic) pulsar.getBrokerService().getTopicReference(topic).get();
+ assertTrue(persistentTopic.getSubscribeRateLimiter().isPresent());
+ });
+
+ try {
+ compactor.compact(topic).get(30, TimeUnit.SECONDS);
+ } finally {
+ admin.namespaces().removeSubscribeRate(namespace);
+ }
+ }
+
@Test
public void testCompaction() throws Exception {
String topic = "persistent://my-tenant/my-ns/compaction";
(pulsar) 07/17: [fix][fn] Make exclusiveLeaderProducer volatile in FunctionMetaDataManager (#26046)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 80e00b29105524781b2e064de0421427f2e355b0
Author: SongOf <46...@users.noreply.github.com>
AuthorDate: Thu Jun 18 06:41:18 2026 +0800
[fix][fn] Make exclusiveLeaderProducer volatile in FunctionMetaDataManager (#26046)
Co-authored-by: maxlisongsong <ma...@didiglobal.com>
(cherry picked from commit b8b1e2c57bd7145fc77ac3c5232957490aee9b77)
---
.../org/apache/pulsar/functions/worker/FunctionMetaDataManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java
index 944b4e8e3449..92680c60c3e7 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java
@@ -72,7 +72,7 @@ public class FunctionMetaDataManager implements AutoCloseable {
// The producer of the metadata topic when we are the leader.
// Note that this variable serves a double duty. A non-null value
// implies we are the leader, while a null value means we are not the leader
- private Producer exclusiveLeaderProducer;
+ private volatile Producer exclusiveLeaderProducer;
@Getter
private volatile MessageId lastMessageSeen = MessageId.earliest;
(pulsar) 06/17: [improve][fn] Upgrade pulsar-client-python to 3.12.0 (#26033)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 75f2411101bb8f5134609e469f469fcd60d571b9
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Tue Jun 16 11:23:00 2026 +0300
[improve][fn] Upgrade pulsar-client-python to 3.12.0 (#26033)
(cherry picked from commit cd7738b4b62c84fd60b2228bdda4956db29f244f)
---
docker/pulsar/Dockerfile | 2 +-
pom.xml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
index fc38c14bbbe1..67e2062a7214 100644
--- a/docker/pulsar/Dockerfile
+++ b/docker/pulsar/Dockerfile
@@ -112,7 +112,7 @@ ARG PULSAR_CLIENT_PYTHON_VERSION
RUN pip3 install --break-system-packages --no-cache-dir \
--only-binary \
grpcio==1.78.0 \
- protobuf==6.33.5 \
+ protobuf==6.33.6 \
pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION} \
kazoo
diff --git a/pom.xml b/pom.xml
index 51f0eadd1a5d..60d188e476ba 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,7 +82,7 @@ flexible messaging model and an intuitive client API.</description>
<pulsar.broker.compiler.release>${maven.compiler.target}</pulsar.broker.compiler.release>
<pulsar.client.compiler.release>8</pulsar.client.compiler.release>
- <pulsar.client.python.version>3.10.0</pulsar.client.python.version>
+ <pulsar.client.python.version>3.12.0</pulsar.client.python.version>
<IMAGE_JDK_MAJOR_VERSION>21</IMAGE_JDK_MAJOR_VERSION>