You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/06/10 12:28:25 UTC

(pulsar) branch branch-4.0 updated (3b83827ccc8 -> de43a840ee1)

This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a change to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git


    from 3b83827ccc8 Bump version to next snapshot version
     new 2cf0aa763c2 [fix][broker] Fix tableview divergence in ServiceUnitStateTableViewSyncer causing flaky tests (#25946)
     new 136bc884c21 [fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState (#25976)
     new 476881fab2d [fix][test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership (#25977)
     new 72e4d264820 [fix][test] Deflake TopicPoliciesTest.setupTestTopic by retrying forced namespace deletion (#25974)
     new de43a840ee1 [improve][misc] Upgrade Apache Commons libraries and Apache Http components (#25963)

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 distribution/server/src/assemble/LICENSE.bin.txt   |  14 +-
 distribution/shell/src/assemble/LICENSE.bin.txt    |   6 +-
 pom.xml                                            |  14 +-
 .../channel/ServiceUnitStateTableViewSyncer.java   | 188 +++++++++---
 .../pulsar/broker/admin/TopicPoliciesTest.java     |  15 +-
 .../ExtensibleLoadManagerImplBaseTest.java         |  75 ++++-
 .../extensions/ExtensibleLoadManagerImplTest.java  | 321 ++++++++++++++-------
 7 files changed, 456 insertions(+), 177 deletions(-)


(pulsar) 04/05: [fix][test] Deflake TopicPoliciesTest.setupTestTopic by retrying forced namespace deletion (#25974)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 72e4d26482026574c5a2346162984bc24649debc
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Jun 8 23:42:01 2026 +0300

    [fix][test] Deflake TopicPoliciesTest.setupTestTopic by retrying forced namespace deletion (#25974)
---
 .../org/apache/pulsar/broker/admin/TopicPoliciesTest.java | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
index 0a1c76ff226..3621c9f21d5 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java
@@ -175,16 +175,11 @@ public class TopicPoliciesTest extends MockedPulsarServiceBaseTest {
         } catch (PulsarAdminException.NotFoundException e) {
             // topic may already be deleted
         }
-        try {
-            admin.namespaces().deleteNamespace(myNamespace, true);
-        } catch (PulsarAdminException.NotFoundException e) {
-            // namespace may already be deleted
-        }
-        try {
-            admin.namespaces().deleteNamespace(myNamespaceV1, true);
-        } catch (PulsarAdminException.NotFoundException e) {
-            // namespace may already be deleted
-        }
+        // Use deleteNamespaceWithRetry since the forced namespace deletion can fail transiently with HTTP 422 when a
+        // topic deletion in the cascade races with concurrent topic loading; the helper retries and treats an
+        // already-deleted namespace as success.
+        deleteNamespaceWithRetry(myNamespace, true);
+        deleteNamespaceWithRetry(myNamespaceV1, true);
         admin.namespaces().createNamespace(testTenant + "/" + testNamespace, Set.of("test"));
         admin.namespaces().createNamespace(myNamespaceV1);
         admin.topics().createPartitionedTopic(testTopic, testTopicPartitions);


(pulsar) 05/05: [improve][misc] Upgrade Apache Commons libraries and Apache Http components (#25963)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit de43a840ee1ea8c4709d9deda1e5ecd5489f1ef9
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Jun 8 16:18:11 2026 +0300

    [improve][misc] Upgrade Apache Commons libraries and Apache Http components (#25963)
    
    (cherry picked from commit b686e96ae873ad59855e44312ff936e856c855a4)
---
 distribution/server/src/assemble/LICENSE.bin.txt | 14 +++++++-------
 distribution/shell/src/assemble/LICENSE.bin.txt  |  6 +++---
 pom.xml                                          | 14 +++++++-------
 3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt
index 7d3f3cb20ab..da1ddf3704c 100644
--- a/distribution/server/src/assemble/LICENSE.bin.txt
+++ b/distribution/server/src/assemble/LICENSE.bin.txt
@@ -284,14 +284,14 @@ The Apache Software License, Version 2.0
  * Apache Commons
     - commons-beanutils-commons-beanutils-1.11.0.jar
     - commons-cli-commons-cli-1.11.0.jar
-    - commons-codec-commons-codec-1.20.0.jar
-    - commons-io-commons-io-2.21.0.jar
-    - commons-logging-commons-logging-1.3.5.jar
+    - commons-codec-commons-codec-1.22.0.jar
+    - commons-io-commons-io-2.22.0.jar
+    - commons-logging-commons-logging-1.3.6.jar
     - org.apache.commons-commons-collections4-4.5.0.jar
     - org.apache.commons-commons-compress-1.28.0.jar
-    - org.apache.commons-commons-configuration2-2.15.0.jar
+    - org.apache.commons-commons-configuration2-2.15.1.jar
     - org.apache.commons-commons-lang3-3.19.0.jar
-    - org.apache.commons-commons-text-1.14.0.jar
+    - org.apache.commons-commons-text-1.15.0.jar
  * Netty
     - io.netty-netty-buffer-4.1.135.Final.jar
     - io.netty-netty-codec-4.1.135.Final.jar
@@ -384,8 +384,8 @@ The Apache Software License, Version 2.0
     - org.apache.bookkeeper-native-io-4.17.3.jar
     - at.yawk.lz4-lz4-java-1.10.3.jar
   * Apache HTTP Client
-    - org.apache.httpcomponents-httpclient-4.5.13.jar
-    - org.apache.httpcomponents-httpcore-4.4.15.jar
+    - org.apache.httpcomponents-httpclient-4.5.14.jar
+    - org.apache.httpcomponents-httpcore-4.4.16.jar
  * AirCompressor
     - io.airlift-aircompressor-2.0.3.jar
  * AsyncHttpClient
diff --git a/distribution/shell/src/assemble/LICENSE.bin.txt b/distribution/shell/src/assemble/LICENSE.bin.txt
index 508d844babb..2bae377ffc9 100644
--- a/distribution/shell/src/assemble/LICENSE.bin.txt
+++ b/distribution/shell/src/assemble/LICENSE.bin.txt
@@ -339,10 +339,10 @@ The Apache Software License, Version 2.0
     - memory-0.8.3.jar
     - sketches-core-0.8.3.jar
  * Apache Commons
-    - commons-codec-1.20.0.jar
-    - commons-io-2.21.0.jar
+    - commons-codec-1.22.0.jar
+    - commons-io-2.22.0.jar
     - commons-lang3-3.19.0.jar
-    - commons-text-1.14.0.jar
+    - commons-text-1.15.0.jar
     - commons-compress-1.28.0.jar
  * Netty
     - netty-buffer-4.1.135.Final.jar
diff --git a/pom.xml b/pom.xml
index efedfd20cb1..51f0eadd1a5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -180,7 +180,7 @@ flexible messaging model and an intuitive client API.</description>
     <bookkeeper.version>4.17.3</bookkeeper.version>
     <zookeeper.version>3.9.5</zookeeper.version>
     <commons-cli.version>1.11.0</commons-cli.version>
-    <commons-text.version>1.14.0</commons-text.version>
+    <commons-text.version>1.15.0</commons-text.version>
     <snappy.version>1.1.10.8</snappy.version> <!-- ZooKeeper server -->
     <dropwizardmetrics.version>4.1.12.1</dropwizardmetrics.version> <!-- ZooKeeper server -->
     <curator.version>5.7.1</curator.version>
@@ -264,9 +264,9 @@ flexible messaging model and an intuitive client API.</description>
     <aircompressor.version>2.0.3</aircompressor.version>
     <asynchttpclient.version>2.15.0</asynchttpclient.version>
     <commons-lang3.version>3.19.0</commons-lang3.version>
-    <commons-io.version>2.21.0</commons-io.version>
-    <commons-codec.version>1.20.0</commons-codec.version>
-    <commons-logging.version>1.3.5</commons-logging.version>
+    <commons-io.version>2.22.0</commons-io.version>
+    <commons-codec.version>1.22.0</commons-codec.version>
+    <commons-logging.version>1.3.6</commons-logging.version>
     <jakarta.ws.rs-api.version>2.1.6</jakarta.ws.rs-api.version>
     <hdrHistogram.version>2.1.9</hdrHistogram.version>
     <javax.servlet-api>3.1.0</javax.servlet-api>
@@ -292,8 +292,8 @@ flexible messaging model and an intuitive client API.</description>
     <nsq-client.version>1.0</nsq-client.version>
     <cron-utils.version>9.1.6</cron-utils.version>
     <spring.version>6.2.12</spring.version>
-    <apache-http-client.version>4.5.13</apache-http-client.version>
-    <apache-httpcomponents.version>4.4.15</apache-httpcomponents.version>
+    <apache-http-client.version>4.5.14</apache-http-client.version>
+    <apache-httpcomponents.version>4.4.16</apache-httpcomponents.version>
     <jetcd.version>0.7.7</jetcd.version>
     <oxia.version>0.7.2</oxia.version>
     <snakeyaml.version>2.0</snakeyaml.version>
@@ -371,7 +371,7 @@ flexible messaging model and an intuitive client API.</description>
     <checkerframework.version>3.33.0</checkerframework.version>
     <nimbus-jose-jwt.version>9.37.4</nimbus-jose-jwt.version>
     <commons-beanutils.version>1.11.0</commons-beanutils.version>
-    <commons-configuration2.version>2.15.0</commons-configuration2.version>
+    <commons-configuration2.version>2.15.1</commons-configuration2.version>
     <mina-core.version>2.1.10</mina-core.version>
     <lz4java.version>1.10.3</lz4java.version>
   </properties>


(pulsar) 01/05: [fix][broker] Fix tableview divergence in ServiceUnitStateTableViewSyncer causing flaky tests (#25946)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 2cf0aa763c21f3b38bc008552c6773f184f9fdb2
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Jun 7 23:10:15 2026 +0300

    [fix][broker] Fix tableview divergence in ServiceUnitStateTableViewSyncer causing flaky tests (#25946)
---
 .../channel/ServiceUnitStateTableViewSyncer.java   | 188 +++++++++---
 .../ExtensibleLoadManagerImplBaseTest.java         |  16 +-
 .../extensions/ExtensibleLoadManagerImplTest.java  | 320 ++++++++++++++-------
 3 files changed, 376 insertions(+), 148 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateTableViewSyncer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateTableViewSyncer.java
index 45ff0dcb267..999c7bea93a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateTableViewSyncer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateTableViewSyncer.java
@@ -23,20 +23,26 @@ import static org.apache.pulsar.broker.ServiceConfiguration.ServiceUnitTableView
 import static org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.COMPACTION_THRESHOLD;
 import static org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.configureSystemTopics;
 import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectWriter;
+import com.google.common.annotations.VisibleForTesting;
 import java.io.Closeable;
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Optional;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+import java.util.function.BiConsumer;
 import lombok.Cleanup;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.common.util.FutureUtil;
 import org.apache.pulsar.common.util.ObjectMapperFactory;
+import org.jspecify.annotations.NonNull;
 
 /**
  * Defines ServiceUnitTableViewSyncer.
@@ -47,10 +53,15 @@ import org.apache.pulsar.common.util.ObjectMapperFactory;
 public class ServiceUnitStateTableViewSyncer implements Closeable {
     private static final int MAX_CONCURRENT_SYNC_COUNT = 100;
     private static final int SYNC_WAIT_TIME_IN_SECS = 300;
+    private static final long RECONCILE_INTERVAL_IN_MILLIS = 5_000;
+    private static final BiConsumer<String, ServiceUnitStateData> NOOP_CONSUMER = (__, ___) -> {
+    };
+    private volatile int syncWaitTimeInSecs = SYNC_WAIT_TIME_IN_SECS;
     private PulsarService pulsar;
     private volatile ServiceUnitStateTableView systemTopicTableView;
     private volatile ServiceUnitStateTableView metadataStoreTableView;
     private volatile boolean isActive = false;
+    private final ObjectWriter jsonWriter = ObjectMapperFactory.getMapper().writer();
 
 
     public void start(PulsarService pulsar)
@@ -82,53 +93,77 @@ public class ServiceUnitStateTableViewSyncer implements Closeable {
     }
 
     private CompletableFuture<Void> syncToSystemTopic(String key, ServiceUnitStateData data) {
-        return systemTopicTableView.put(key, data);
+        return logIfFailed(sync(systemTopicTableView, key, data), key, data, "systemTopic");
     }
 
     private CompletableFuture<Void> syncToMetadataStore(String key, ServiceUnitStateData data) {
-        return metadataStoreTableView.put(key, data);
+        return logIfFailed(sync(metadataStoreTableView, key, data), key, data, "metadataStore");
     }
 
-    private void dummy(String key, ServiceUnitStateData data) {
+    private CompletableFuture<Void> sync(ServiceUnitStateTableView dst, String key, ServiceUnitStateData data) {
+        // A null tail item is a tombstone: the source view removed the key. Route it to
+        // delete() rather than put(): the metadata-store view's put() rejects null
+        // (@NonNull) and the system-topic view's delete() is itself a null-valued put(),
+        // so a uniform delete keeps both sync directions symmetric and prevents a missed
+        // deletion from leaving the two views with different sizes (which would make
+        // waitUntilSynced spin until the timeout budget).
+        return data == null ? dst.delete(key) : dst.put(key, data);
+    }
+
+    private CompletableFuture<Void> logIfFailed(CompletableFuture<Void> future, String key,
+                                                ServiceUnitStateData data, String dst) {
+        return future.whenComplete((__, e) -> {
+            if (e != null && !(e instanceof PulsarClientException.AlreadyClosedException)) {
+                log.warn("Failed to sync tableview item; sizes may diverge until the next update;"
+                        + " dst={} serviceUnit={} data={}", dst, key, data, e);
+            }
+        });
     }
 
     private void syncExistingItems()
             throws IOException, ExecutionException, InterruptedException, TimeoutException {
         long started = System.currentTimeMillis();
+
         @Cleanup
         ServiceUnitStateTableView metadataStoreTableView = new ServiceUnitStateMetadataStoreTableViewImpl();
         metadataStoreTableView.start(
                 pulsar,
-                this::dummy,
-                this::dummy,
-                this::dummy
+                NOOP_CONSUMER,
+                NOOP_CONSUMER,
+                NOOP_CONSUMER
         );
 
         @Cleanup
         ServiceUnitStateTableView systemTopicTableView = new ServiceUnitStateTableViewImpl();
         systemTopicTableView.start(
                 pulsar,
-                this::dummy,
-                this::dummy,
-                this::dummy
+                NOOP_CONSUMER,
+                NOOP_CONSUMER,
+                NOOP_CONSUMER
         );
 
 
         var syncer = pulsar.getConfiguration().getLoadBalancerServiceUnitTableViewSyncer();
+        ServiceUnitStateTableView src;
+        ServiceUnitStateTableView dst;
         if (syncer == SystemTopicToMetadataStoreSyncer) {
             clean(metadataStoreTableView);
             syncExistingItemsToMetadataStore(systemTopicTableView);
+            src = systemTopicTableView;
+            dst = metadataStoreTableView;
         } else {
             clean(systemTopicTableView);
             syncExistingItemsToSystemTopic(metadataStoreTableView, systemTopicTableView);
+            src = metadataStoreTableView;
+            dst = systemTopicTableView;
         }
 
-        if (!waitUntilSynced(metadataStoreTableView, systemTopicTableView, started)) {
+        if (!waitUntilSynced(src, dst, started)) {
             throw new TimeoutException(
                     syncer + " failed to sync existing items in tableviews. MetadataStoreTableView.size: "
                             + metadataStoreTableView.entrySet().size()
                             + ", SystemTopicTableView.size: " + systemTopicTableView.entrySet().size() + " in "
-                            + SYNC_WAIT_TIME_IN_SECS + " secs");
+                            + syncWaitTimeInSecs + " secs");
         }
 
         log.info("Synced existing items MetadataStoreTableView.size:{} , "
@@ -154,8 +189,8 @@ public class ServiceUnitStateTableViewSyncer implements Closeable {
         this.metadataStoreTableView.start(
                 pulsar,
                 this::syncToSystemTopic,
-                this::dummy,
-                this::dummy
+                NOOP_CONSUMER,
+                NOOP_CONSUMER
         );
         log.info("Started MetadataStoreTableView");
 
@@ -163,18 +198,20 @@ public class ServiceUnitStateTableViewSyncer implements Closeable {
         this.systemTopicTableView.start(
                 pulsar,
                 this::syncToMetadataStore,
-                this::dummy,
-                this::dummy
+                NOOP_CONSUMER,
+                NOOP_CONSUMER
         );
         log.info("Started SystemTopicTableView");
 
         var syncer = pulsar.getConfiguration().getLoadBalancerServiceUnitTableViewSyncer();
-        if (!waitUntilSynced(metadataStoreTableView, systemTopicTableView, started)) {
+        var src = syncer == SystemTopicToMetadataStoreSyncer ? systemTopicTableView : metadataStoreTableView;
+        var dst = syncer == SystemTopicToMetadataStoreSyncer ? metadataStoreTableView : systemTopicTableView;
+        if (!waitUntilSynced(src, dst, started)) {
             throw new TimeoutException(
                     syncer + " failed to sync tableviews. MetadataStoreTableView.size: "
                             + metadataStoreTableView.entrySet().size()
                             + ", SystemTopicTableView.size: " + systemTopicTableView.entrySet().size() + " in "
-                            + SYNC_WAIT_TIME_IN_SECS + " secs");
+                            + syncWaitTimeInSecs + " secs");
         }
 
 
@@ -187,62 +224,134 @@ public class ServiceUnitStateTableViewSyncer implements Closeable {
     private void syncExistingItemsToMetadataStore(ServiceUnitStateTableView src)
             throws JsonProcessingException, ExecutionException, InterruptedException, TimeoutException {
         // Directly use store to sync existing items to metadataStoreTableView(otherwise, they are conflicted out)
-        var store = pulsar.getLocalMetadataStore();
-        var writer = ObjectMapperFactory.getMapper().writer();
-        var opTimeout = pulsar.getConfiguration().getMetadataStoreOperationTimeoutSeconds();
         List<CompletableFuture<Void>> futures = new ArrayList<>();
         var srcIter = src.entrySet().iterator();
         while (srcIter.hasNext()) {
             var e = srcIter.next();
-            futures.add(store.put(ServiceUnitStateMetadataStoreTableViewImpl.PATH_PREFIX + "/" + e.getKey(),
-                    writer.writeValueAsBytes(e.getValue()), Optional.empty()).thenApply(__ -> null));
-            if (futures.size() == MAX_CONCURRENT_SYNC_COUNT || !srcIter.hasNext()) {
-                FutureUtil.waitForAll(futures).get(opTimeout, TimeUnit.SECONDS);
-            }
+            futures.add(writeToMetadataStore(e.getKey(), e.getValue()));
+            maybeWaitCompletion(futures, !srcIter.hasNext());
+        }
+    }
+
+    private void maybeWaitCompletion(List<CompletableFuture<Void>> futures, boolean forceWait)
+            throws InterruptedException, ExecutionException, TimeoutException {
+        if (!futures.isEmpty() && (futures.size() == MAX_CONCURRENT_SYNC_COUNT || forceWait)) {
+            FutureUtil.waitForAll(futures)
+                    .get(pulsar.getConfiguration().getMetadataStoreOperationTimeoutSeconds(), TimeUnit.SECONDS);
+            futures.clear();
         }
     }
 
+    private @NonNull CompletableFuture<Void> writeToMetadataStore(String key, ServiceUnitStateData value)
+            throws JsonProcessingException {
+        return pulsar.getLocalMetadataStore().put(ServiceUnitStateMetadataStoreTableViewImpl.PATH_PREFIX + "/" + key,
+                jsonWriter.writeValueAsBytes(value), Optional.empty()).thenApply(__ -> null);
+    }
+
     private void syncExistingItemsToSystemTopic(ServiceUnitStateTableView src,
                                                 ServiceUnitStateTableView dst)
             throws ExecutionException, InterruptedException, TimeoutException {
-        var opTimeout = pulsar.getConfiguration().getMetadataStoreOperationTimeoutSeconds();
         List<CompletableFuture<Void>> futures = new ArrayList<>();
         var srcIter = src.entrySet().iterator();
         while (srcIter.hasNext()) {
             var e = srcIter.next();
             futures.add(dst.put(e.getKey(), e.getValue()));
-            if (futures.size() == MAX_CONCURRENT_SYNC_COUNT || !srcIter.hasNext()) {
-                FutureUtil.waitForAll(futures).get(opTimeout, TimeUnit.SECONDS);
-            }
+            maybeWaitCompletion(futures, !srcIter.hasNext());
         }
     }
 
     private void clean(ServiceUnitStateTableView dst)
             throws ExecutionException, InterruptedException, TimeoutException {
-        var opTimeout = pulsar.getConfiguration().getMetadataStoreOperationTimeoutSeconds();
         var dstIter = dst.entrySet().iterator();
         List<CompletableFuture<Void>> futures = new ArrayList<>();
         while (dstIter.hasNext()) {
             var e = dstIter.next();
             futures.add(dst.delete(e.getKey()));
-            if (futures.size() == MAX_CONCURRENT_SYNC_COUNT || !dstIter.hasNext()) {
-                FutureUtil.waitForAll(futures).get(opTimeout, TimeUnit.SECONDS);
-            }
+            maybeWaitCompletion(futures, !dstIter.hasNext());
         }
     }
 
-    private boolean waitUntilSynced(ServiceUnitStateTableView srt, ServiceUnitStateTableView dst, long started)
+    private boolean waitUntilSynced(ServiceUnitStateTableView src, ServiceUnitStateTableView dst, long started)
             throws InterruptedException {
-        while (srt.entrySet().size() != dst.entrySet().size()) {
-            if (TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - started)
-                    > SYNC_WAIT_TIME_IN_SECS) {
+        long lastReconciled = started;
+        while (src.entrySet().size() != dst.entrySet().size()) {
+            long now = System.currentTimeMillis();
+            if (TimeUnit.MILLISECONDS.toSeconds(now - started) > syncWaitTimeInSecs) {
                 return false;
             }
+            // Give in-flight syncs a grace period to settle on their own, then reconcile
+            // periodically: updates that raced with the table views' (re)start were replayed
+            // to the fresh views as existing items — which are deliberately not wired to
+            // sync — so without reconciliation the views would never converge.
+            if (now - lastReconciled >= RECONCILE_INTERVAL_IN_MILLIS) {
+                if (log.isDebugEnabled()) {
+                    log.debug("Tableviews not synced yet; reconciling; srcSize={} dstSize={} elapsedSecs={}",
+                            src.entrySet().size(), dst.entrySet().size(),
+                            TimeUnit.MILLISECONDS.toSeconds(now - started));
+                }
+                reconcile(src, dst, started);
+                lastReconciled = now;
+            }
             Thread.sleep(100);
         }
         return true;
     }
 
+    /**
+     * Copies items the destination table view is missing and removes stale items that no longer
+     * exist in the source. Channel updates that land between the existing-items copy and the
+     * registration of the tail listeners are only visible as existing items of the freshly
+     * started views, so the tail listeners never see them. Writes flow to the migration source
+     * while the syncer starts, making the source view authoritative; destination-only items are
+     * removed only when they predate this sync phase and are still absent from the source, so a
+     * concurrent fresh write to the destination is never discarded. Failures are logged and left
+     * for the next reconcile pass. Runs on the caller's (load manager) thread with each batch
+     * bounded by the metadata store operation timeout.
+     */
+    private void reconcile(ServiceUnitStateTableView src, ServiceUnitStateTableView dst, long started)
+            throws InterruptedException {
+        // Snapshot the destination entries before iterating the source so that a key arriving
+        // in the destination through a concurrent tail sync cannot be misclassified as stale.
+        var staleDstItems = new HashMap<String, ServiceUnitStateData>();
+        for (var e : dst.entrySet()) {
+            staleDstItems.put(e.getKey(), e.getValue());
+        }
+        try {
+            List<CompletableFuture<Void>> futures = new ArrayList<>();
+            for (var e : src.entrySet()) {
+                if (staleDstItems.remove(e.getKey()) == null) {
+                    log.info("Reconciling item missing from the destination tableview; serviceUnit={}",
+                            e.getKey());
+                    if (dst.isMetadataStoreBased()) {
+                        // Write directly to the store like syncExistingItemsToMetadataStore
+                        // does; the view's put() would conflict the item out.
+                        futures.add(writeToMetadataStore(e.getKey(), e.getValue()));
+                    } else {
+                        futures.add(dst.put(e.getKey(), e.getValue()));
+                    }
+                    maybeWaitCompletion(futures, false);
+                }
+            }
+            for (var e : staleDstItems.entrySet()) {
+                // Only remove items written before this sync phase began and re-confirmed absent
+                // from the source: a fresh destination write (e.g. from a broker already switched
+                // to the destination implementation) is propagated to the source by the tail
+                // listener instead of being deleted here.
+                if (e.getValue().timestamp() < started && src.get(e.getKey()) == null) {
+                    log.info("Reconciling stale item in the destination tableview; serviceUnit={}",
+                            e.getKey());
+                    futures.add(dst.delete(e.getKey()));
+                    maybeWaitCompletion(futures, false);
+                }
+            }
+            maybeWaitCompletion(futures, true);
+        } catch (IOException | ExecutionException | TimeoutException e) {
+            // Transient write failures leave a size divergence behind; the next reconcile pass
+            // (or the sync-wait timeout) handles it.
+            log.warn("Failed to reconcile tableview items", e);
+        }
+    }
+
     @Override
     public void close() throws IOException {
         if (!isActive) {
@@ -282,4 +391,9 @@ public class ServiceUnitStateTableViewSyncer implements Closeable {
     public boolean isActive() {
         return isActive;
     }
+
+    @VisibleForTesting
+    public void setSyncWaitTimeInSecs(int syncWaitTimeInSecs) {
+        this.syncWaitTimeInSecs = syncWaitTimeInSecs;
+    }
 }
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
index 55e9b8d6baf..2b30723f0a2 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
@@ -47,6 +47,7 @@ import org.apache.pulsar.common.naming.SystemTopicNames;
 import org.apache.pulsar.common.naming.TopicName;
 import org.apache.pulsar.common.policies.data.ClusterData;
 import org.apache.pulsar.common.policies.data.TenantInfoImpl;
+import org.awaitility.Awaitility;
 import org.testng.annotations.AfterClass;
 import org.testng.annotations.BeforeClass;
 import org.testng.annotations.BeforeMethod;
@@ -196,7 +197,20 @@ public abstract class ExtensibleLoadManagerImplBaseTest extends MockedPulsarServ
 
     @BeforeMethod(alwaysRun = true)
     protected void initializeState() throws PulsarAdminException, IllegalAccessException {
-        admin.namespaces().unload(defaultTestNamespace);
+        // After a prior test churned leader election, the channel-topic bundle can be left
+        // unserved ("not served by this instance"), making the unload's channel publish fail
+        // (HTTP 500) or hang server-side until the background monitor task (120s interval)
+        // reconciles the brokers' roles with the channel ownership. Drive monitor() eagerly to
+        // heal that state, bound each unload attempt (a synchronous unload() can block longer
+        // than the whole retry window), and fail loudly on exhaustion.
+        Awaitility.await().atMost(120, TimeUnit.SECONDS)
+                .pollInterval(1, TimeUnit.SECONDS)
+                .ignoreExceptions()
+                .untilAsserted(() -> {
+                    primaryLoadManager.monitor();
+                    secondaryLoadManager.monitor();
+                    admin.namespaces().unloadAsync(defaultTestNamespace).get(15, TimeUnit.SECONDS);
+                });
         reset(primaryLoadManager, secondaryLoadManager);
         FieldUtils.writeDeclaredField(pulsarClient, "lookup", lookupService, true);
         pulsar1.getConfig().setLoadBalancerMultiPhaseBundleUnload(true);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
index 65d017499fd..05e9bfba6ef 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
@@ -1331,7 +1331,11 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         }
     }
 
-    @Test(priority = 200)
+    // Cap below the 300s suite default (AnnotationListener) so a hung ServiceUnitStateTableViewSyncer
+    // start() fails fast and is retried instead of consuming the full 300s slot and corrupting the
+    // next @BeforeMethod. Combined with the shortened sync-wait budget set below, a real divergence
+    // surfaces within the shortened budget instead of a 5-minute ThreadTimeoutException.
+    @Test(priority = 200, timeOut = 240 * 1000)
     public void testLoadBalancerServiceUnitTableViewSyncer() throws Exception {
         // Make pulsar1 the leader so primaryLoadManager is the syncer-running broker.
         makePrimaryAsLeader();
@@ -1361,120 +1365,142 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         String syncerType =
                 serviceUnitStateTableViewClassName.equals(ServiceUnitStateTableViewImpl.class.getName())
                         ? "SystemTopicToMetadataStoreSyncer" : "MetadataStoreToSystemTopicSyncer";
+        // Shrink the sync-wait budget on both brokers' live syncers BEFORE the first start()
+        // (driven by monitor() below) so any tableview-size divergence fails in ~30s with the
+        // syncer's own TimeoutException instead of spinning for the full 300s default — the
+        // exact hang observed in CI happened inside that first start().
+        primaryLoadManager.getServiceUnitStateTableViewSyncer().setSyncWaitTimeInSecs(30);
+        secondaryLoadManager.getServiceUnitStateTableViewSyncer().setSyncWaitTimeInSecs(30);
+
         pulsar.getAdminClient().brokers()
                 .updateDynamicConfiguration("loadBalancerServiceUnitTableViewSyncer", syncerType);
         Awaitility.await().untilAsserted(() ->
                 assertTrue(pulsar1.getConfiguration().isLoadBalancerServiceUnitTableViewSyncerEnabled()));
-        primaryLoadManager.monitor();
-        Awaitility.await().atMost(30, TimeUnit.SECONDS)
-                .untilAsserted(() -> assertTrue(primaryLoadManager.getServiceUnitStateTableViewSyncer()
-                        .isActive()));
-        assertFalse(secondaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
-
-        // === Phase 2: add a 3rd broker using the OTHER table view impl ===
-        // pulsar1/pulsar2 use serviceUnitStateTableViewClassName; pulsar3 deliberately
-        // uses the other one so the test exercises cross-impl lookups regardless of
-        // which parametrization we're running.
-        String otherClassName =
-                serviceUnitStateTableViewClassName.equals(ServiceUnitStateTableViewImpl.class.getName())
-                        ? ServiceUnitStateMetadataStoreTableViewImpl.class.getName()
-                        : ServiceUnitStateTableViewImpl.class.getName();
-
-        ServiceConfiguration crossImplConf = getDefaultConf();
-        crossImplConf.setAllowAutoTopicCreation(true);
-        crossImplConf.setForceDeleteNamespaceAllowed(true);
-        crossImplConf.setLoadManagerClassName(ExtensibleLoadManagerImpl.class.getCanonicalName());
-        crossImplConf.setLoadBalancerLoadSheddingStrategy(TransferShedder.class.getName());
-        crossImplConf.setLoadManagerServiceUnitStateTableViewClassName(otherClassName);
-
-        try (var crossImplCtx = createAdditionalPulsarTestContext(crossImplConf)) {
-            var pulsar3 = crossImplCtx.getPulsarService();
-
-            // All three brokers (across both impls) must agree on topic ownership.
-            assertEquals(pulsar2.getAdminClient().lookups().lookupTopic(topic), ownershipBefore);
-            assertEquals(pulsar3.getAdminClient().lookups().lookupTopic(topic), ownershipBefore);
-            Optional<URL> webUrlPulsar3 = pulsar3.getNamespaceService().getWebServiceUrl(bundle, options);
-            assertTrue(webUrlPulsar3.isPresent());
-            assertEquals(webUrlPulsar3.get().toString(), webUrlBefore.get().toString());
-
-            // SLA monitor and heartbeat lookups must agree across impls in every direction.
-            List<PulsarService> brokers = List.of(pulsar1, pulsar2, pulsar3);
-            for (PulsarService viewer : brokers) {
-                for (PulsarService owner : brokers) {
-                    assertLookupHeartbeatOwner(viewer, owner.getBrokerId(), owner.getBrokerServiceUrl());
-                    assertLookupSLANamespaceOwner(viewer, owner.getBrokerId(), owner.getBrokerServiceUrl());
+        // Drive monitor() inside the await so a transient start() failure (swallowed by
+        // monitor()'s catch) is retried instead of waiting for the 120s background monitor task.
+        Awaitility.await().atMost(120, TimeUnit.SECONDS).untilAsserted(() -> {
+            primaryLoadManager.monitor();
+            assertTrue(primaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
+        });
+
+        try {
+            assertFalse(secondaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
+
+            // === Phase 2: add a 3rd broker using the OTHER table view impl ===
+            // pulsar1/pulsar2 use serviceUnitStateTableViewClassName; pulsar3 deliberately
+            // uses the other one so the test exercises cross-impl lookups regardless of
+            // which parametrization we're running.
+            String otherClassName =
+                    serviceUnitStateTableViewClassName.equals(ServiceUnitStateTableViewImpl.class.getName())
+                            ? ServiceUnitStateMetadataStoreTableViewImpl.class.getName()
+                            : ServiceUnitStateTableViewImpl.class.getName();
+
+            ServiceConfiguration crossImplConf = getDefaultConf();
+            crossImplConf.setAllowAutoTopicCreation(true);
+            crossImplConf.setForceDeleteNamespaceAllowed(true);
+            crossImplConf.setLoadManagerClassName(ExtensibleLoadManagerImpl.class.getCanonicalName());
+            crossImplConf.setLoadBalancerLoadSheddingStrategy(TransferShedder.class.getName());
+            crossImplConf.setLoadManagerServiceUnitStateTableViewClassName(otherClassName);
+
+            try (var crossImplCtx = createAdditionalPulsarTestContext(crossImplConf)) {
+                var pulsar3 = crossImplCtx.getPulsarService();
+
+                // All three brokers (across both impls) must agree on topic ownership.
+                assertEquals(pulsar2.getAdminClient().lookups().lookupTopic(topic), ownershipBefore);
+                assertEquals(pulsar3.getAdminClient().lookups().lookupTopic(topic), ownershipBefore);
+                Optional<URL> webUrlPulsar3 = pulsar3.getNamespaceService().getWebServiceUrl(bundle, options);
+                assertTrue(webUrlPulsar3.isPresent());
+                assertEquals(webUrlPulsar3.get().toString(), webUrlBefore.get().toString());
+
+                // SLA monitor and heartbeat lookups must agree across impls in every direction.
+                List<PulsarService> brokers = List.of(pulsar1, pulsar2, pulsar3);
+                for (PulsarService viewer : brokers) {
+                    for (PulsarService owner : brokers) {
+                        assertLookupHeartbeatOwner(viewer, owner.getBrokerId(), owner.getBrokerServiceUrl());
+                        assertLookupSLANamespaceOwner(viewer, owner.getBrokerId(), owner.getBrokerServiceUrl());
+                    }
                 }
-            }
 
-            // === Phase 3: simulate the cross-impl broker going offline ===
-            // Its SLA namespace must reassign to a remaining broker, and the ownership
-            // change must propagate through the syncer to brokers using the other impl.
-            var wrapper3 = (ExtensibleLoadManagerWrapper) pulsar3.getLoadManager().get();
-            var loadManager3 = (ExtensibleLoadManagerImpl)
-                    FieldUtils.readField(wrapper3, "loadManager", true);
-            ServiceUnitStateChannel channel3 = (ServiceUnitStateChannel)
-                    FieldUtils.readField(loadManager3, "serviceUnitStateChannel", true);
-            channel3.cleanOwnerships();
-            // Set state to Closed BEFORE deleting the ZK node to prevent the notification
-            // handler's session-expiry recovery from auto-re-registering broker3. In
-            // production the PulsarService shuts down after unregister(), so the handler
-            // never fires; in tests the service stays running and creates a race.
-            var registry3 = (BrokerRegistryImpl) loadManager3.getBrokerRegistry();
-            registry3.state.set(BrokerRegistryImpl.State.Closed);
-            pulsar3.getLocalMetadataStore()
-                    .delete("/loadbalance/brokers/" + pulsar3.getBrokerId(), Optional.empty()).get();
-
-            String slaMonitorTopic = getSLAMonitorNamespace(pulsar3.getBrokerId(), pulsar.getConfiguration())
-                    .getPersistentTopicName("test");
-            String pulsar3BrokerUrl = pulsar3.getBrokerServiceUrl();
-            Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {
-                String reassigned = pulsar.getAdminClient().lookups().lookupTopic(slaMonitorTopic);
-                assertNotNull(reassigned);
-                assertNotEquals(reassigned, pulsar3BrokerUrl);
-            });
+                // === Phase 3: simulate the cross-impl broker going offline ===
+                // Its SLA namespace must reassign to a remaining broker, and the ownership
+                // change must propagate through the syncer to brokers using the other impl.
+                var wrapper3 = (ExtensibleLoadManagerWrapper) pulsar3.getLoadManager().get();
+                var loadManager3 = (ExtensibleLoadManagerImpl)
+                        FieldUtils.readField(wrapper3, "loadManager", true);
+                ServiceUnitStateChannel channel3 = (ServiceUnitStateChannel)
+                        FieldUtils.readField(loadManager3, "serviceUnitStateChannel", true);
+                channel3.cleanOwnerships();
+                // Set state to Closed BEFORE deleting the ZK node to prevent the notification
+                // handler's session-expiry recovery from auto-re-registering broker3. In
+                // production the PulsarService shuts down after unregister(), so the handler
+                // never fires; in tests the service stays running and creates a race.
+                var registry3 = (BrokerRegistryImpl) loadManager3.getBrokerRegistry();
+                registry3.state.set(BrokerRegistryImpl.State.Closed);
+                pulsar3.getLocalMetadataStore()
+                        .delete("/loadbalance/brokers/" + pulsar3.getBrokerId(), Optional.empty()).get();
+
+                String slaMonitorTopic = getSLAMonitorNamespace(pulsar3.getBrokerId(), pulsar.getConfiguration())
+                        .getPersistentTopicName("test");
+                String pulsar3BrokerUrl = pulsar3.getBrokerServiceUrl();
+                Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {
+                    String reassigned = pulsar.getAdminClient().lookups().lookupTopic(slaMonitorTopic);
+                    assertNotNull(reassigned);
+                    assertNotEquals(reassigned, pulsar3BrokerUrl);
+                });
 
-            // Send a message while the topic is owned by the reassigned broker; this must
-            // remain durable when ownership migrates back below.
-            @Cleanup
-            Producer<String> producer = pulsar.getClient().newProducer(Schema.STRING)
-                    .topic(slaMonitorTopic).create();
-            producer.send("offline");
-
-            // === Phase 4: re-register the broker and verify ownership returns ===
-            registry3.state.set(BrokerRegistryImpl.State.Started);
-            registry3.registerAsync().get();
-            Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() ->
-                    assertEquals(pulsar.getAdminClient().lookups().lookupTopic(slaMonitorTopic),
-                            pulsar3.getBrokerServiceUrl()));
-
-            // Same producer reconnects to the new owner; a fresh producer also works.
-            producer.send("after-reconnect");
-            @Cleanup
-            Producer<String> producer2 = pulsar.getClient().newProducer(Schema.STRING)
-                    .topic(slaMonitorTopic).create();
-            producer2.send("from-new-producer");
+                // Send a message while the topic is owned by the reassigned broker; this must
+                // remain durable when ownership migrates back below.
+                @Cleanup
+                Producer<String> producer = pulsar.getClient().newProducer(Schema.STRING)
+                        .topic(slaMonitorTopic).create();
+                producer.send("offline");
 
-            @Cleanup
-            Consumer<String> consumer = pulsar.getClient().newConsumer(Schema.STRING)
-                    .topic(slaMonitorTopic)
-                    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
-                    .subscriptionName("test")
-                    .subscribe();
-            assertEquals(consumer.receive().getValue(), "offline");
-            assertEquals(consumer.receive().getValue(), "after-reconnect");
-            assertEquals(consumer.receive().getValue(), "from-new-producer");
-        }
+                // === Phase 4: re-register the broker and verify ownership returns ===
+                registry3.state.set(BrokerRegistryImpl.State.Started);
+                registry3.registerAsync().get();
+                Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() ->
+                        assertEquals(pulsar.getAdminClient().lookups().lookupTopic(slaMonitorTopic),
+                                pulsar3.getBrokerServiceUrl()));
 
-        // === Phase 5: disable the syncer and verify it deactivates ===
-        pulsar.getAdminClient().brokers()
-                .deleteDynamicConfiguration("loadBalancerServiceUnitTableViewSyncer");
-        Awaitility.await().untilAsserted(() ->
-                assertFalse(pulsar1.getConfiguration().isLoadBalancerServiceUnitTableViewSyncerEnabled()));
-        primaryLoadManager.monitor();
-        Awaitility.await().atMost(30, TimeUnit.SECONDS)
-                .untilAsserted(() -> assertFalse(primaryLoadManager.getServiceUnitStateTableViewSyncer()
-                        .isActive()));
-        assertFalse(secondaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
+                // Same producer reconnects to the new owner; a fresh producer also works.
+                producer.send("after-reconnect");
+                @Cleanup
+                Producer<String> producer2 = pulsar.getClient().newProducer(Schema.STRING)
+                        .topic(slaMonitorTopic).create();
+                producer2.send("from-new-producer");
+
+                @Cleanup
+                Consumer<String> consumer = pulsar.getClient().newConsumer(Schema.STRING)
+                        .topic(slaMonitorTopic)
+                        .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
+                        .subscriptionName("test")
+                        .subscribe();
+                assertEquals(consumer.receive().getValue(), "offline");
+                assertEquals(consumer.receive().getValue(), "after-reconnect");
+                assertEquals(consumer.receive().getValue(), "from-new-producer");
+            }
+        } finally {
+            // === Phase 5: disable the syncer and verify it deactivates ===
+            // Guarantee the dynamic config is removed and the syncer is driven inactive even if
+            // the body threw, so the syncer cannot stay enabled and poison later tests. Note this
+            // cannot tear down a start() that failed before isActive=true (close() short-circuits
+            // on !isActive); leftover tail views from such a partial start are recovered by the
+            // next successful start(), and the next test's initializeState() retry absorbs any
+            // residual channel disruption.
+            try {
+                pulsar.getAdminClient().brokers()
+                        .deleteDynamicConfiguration("loadBalancerServiceUnitTableViewSyncer");
+            } catch (Exception e) {
+                log.warn("Failed to delete syncer dynamic config in cleanup", e);
+            }
+            Awaitility.await().atMost(60, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {
+                assertFalse(pulsar1.getConfiguration().isLoadBalancerServiceUnitTableViewSyncerEnabled());
+                primaryLoadManager.monitor();
+                secondaryLoadManager.monitor();
+                assertFalse(primaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
+                assertFalse(secondaryLoadManager.getServiceUnitStateTableViewSyncer().isActive());
+            });
+        }
     }
 
     private void assertLookupHeartbeatOwner(PulsarService pulsar,
@@ -1541,7 +1567,52 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         });
     }
 
-    @Test(timeOut = 30 * 1000, priority = 2100)
+    // After a test churns leader election, the channel-topic bundle can be transiently
+    // unowned and the channel producer can be in reconnect backoff. The next @BeforeMethod
+    // (initializeState -> namespaces().unload(...)) publishes a state change on the channel
+    // topic; if it runs in that window the producer send times out (HTTP 500). Give the
+    // re-election a best-effort chance to settle before yielding to the next test.
+    //
+    // This is a best-effort smoothing wait, not an assertion: the budget is deliberately a
+    // fraction (20s) of the callers' 60s method timeout so it cannot consume the whole slot
+    // and trip TestNG's ThreadTimeoutException mid-poll, and any failure to settle
+    // is swallowed-and-logged rather than thrown. That matters because callers invoke this from
+    // a finally block — a settling delay here must never replace (mask) the body's exception.
+    // The next test's initializeState() carries a 60s ignoreExceptions retry as the real backstop.
+    private void awaitChannelOwnerStable() {
+        try {
+            Awaitility.await().atMost(20, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {
+                // monitor() reconciles each broker's role with the channel ownership and
+                // re-serves the channel-topic bundle if the leadership churn left it unserved
+                // ("not served by this instance") — the same self-healing the 120s background
+                // monitor task provides, driven eagerly so the next test does not start inside
+                // the broken window.
+                primaryLoadManager.monitor();
+                secondaryLoadManager.monitor();
+                Optional<String> owner1 = channel1.getChannelOwnerAsync().get(5, TimeUnit.SECONDS);
+                Optional<String> owner2 = channel2.getChannelOwnerAsync().get(5, TimeUnit.SECONDS);
+                assertTrue(owner1.isPresent());
+                assertEquals(owner1, owner2);
+                assertTrue(channel1.isChannelOwner() ^ channel2.isChannelOwner());
+                // Probe that the channel topic is actually served: the lookup re-assigns the
+                // pulsar/system bundle if it is unowned, and getStats proves the owner loads
+                // the topic (the lookup layer alone can claim an owner that refuses to serve).
+                String channelTopic = ServiceUnitStateTableViewImpl.TOPIC;
+                assertNotNull(pulsar.getAdminClient().lookups().lookupTopic(channelTopic));
+                if (serviceUnitStateTableViewClassName.equals(
+                        ServiceUnitStateTableViewImpl.class.getName())) {
+                    assertNotNull(pulsar.getAdminClient().topics().getStats(channelTopic));
+                }
+            });
+        } catch (Throwable t) {
+            log.warn("Channel owner did not stabilize within the best-effort window; "
+                    + "relying on the next test's initializeState() retry", t);
+        }
+    }
+
+    // 60s: the body's repeated role transitions plus the trailing awaitChannelOwnerStable() can
+    // exceed 30s under load.
+    @Test(timeOut = 60 * 1000, priority = 2100)
     public void testRoleChangeIdempotency() throws Exception {
 
         makePrimaryAsLeader();
@@ -1621,7 +1692,8 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
             assertEquals(ExtensibleLoadManagerImpl.Role.Follower,
                     secondaryLoadManager.getRole());
 
-
+            // Confirm a stable channel owner before yielding to the next test's @BeforeMethod.
+            awaitChannelOwnerStable();
     }
 
     @DataProvider(name = "noChannelOwnerMonitorHandler")
@@ -1629,7 +1701,9 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         return new Object[][] { { true }, { false } };
     }
 
-    @Test(dataProvider = "noChannelOwnerMonitorHandler", timeOut = 30 * 1000, priority = 2101)
+    // 60s: the body's leader-election churn plus the trailing awaitChannelOwnerStable() can
+    // exceed 30s under load.
+    @Test(dataProvider = "noChannelOwnerMonitorHandler", timeOut = 60 * 1000, priority = 2101)
     public void testHandleNoChannelOwner(boolean noChannelOwnerMonitorHandler) throws Exception {
 
         makePrimaryAsLeader();
@@ -1696,10 +1770,25 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
             // clean up for monitor test
             pulsar1.getLeaderElectionService().start();
             pulsar2.getLeaderElectionService().start();
+            // If the body failed mid-churn, both restarted elections can keep flapping the
+            // leadership between the brokers, leaving the channel-topic bundle unserved
+            // beyond what the next test's initializeState() retry can absorb. Force a
+            // deterministic single leader before stabilizing (best-effort: must not mask
+            // the body's exception).
+            try {
+                makePrimaryAsLeader();
+            } catch (Throwable t) {
+                log.warn("Failed to re-establish primary as leader in cleanup", t);
+            }
+            // Re-establish a stable channel owner before yielding to the next test's
+            // @BeforeMethod, which publishes to the channel topic via namespace unload.
+            awaitChannelOwnerStable();
         }
     }
 
-    @Test(timeOut = 30 * 1000, priority = 2000)
+    // 60s: the body's role transitions plus the trailing awaitChannelOwnerStable() can exceed
+    // 30s under load (observed locally at 30.017s).
+    @Test(timeOut = 60 * 1000, priority = 2000)
     public void testRoleChange() throws Exception {
         makePrimaryAsLeader();
 
@@ -1719,6 +1808,11 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
                 new NamespaceBundleStats()));
 
         Awaitility.await().atMost(30, TimeUnit.SECONDS).untilAsserted(() -> {
+            // The internal topics live in the pulsar/system bundle; if the leadership churn
+            // left it unserved, only a monitor() role reconciliation re-serves it (the
+            // background monitor task would take up to 120s) — drive it while waiting.
+            leader.monitor();
+            follower.monitor();
 
             assertNotNull(FieldUtils.readDeclaredField(leader.getTopBundlesLoadDataStore(),
                     "tableView", true));
@@ -1756,6 +1850,9 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         topBundlesExpected.getTopBundlesLoadData().get(0).stats().msgRateIn = 1;
 
         Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {
+            // Same monitor()-driven healing as above for the post-transfer assertions.
+            leader2.monitor();
+            follower2.monitor();
             assertNotNull(FieldUtils.readDeclaredField(leader2.getTopBundlesLoadDataStore(), "tableView", true));
             assertNull(FieldUtils.readDeclaredField(follower2.getTopBundlesLoadDataStore(), "tableView", true));
 
@@ -1780,6 +1877,9 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
         follower2.getBrokerLoadDataStore().pushAsync(key, brokerLoadExpected).get(3, TimeUnit.SECONDS);
         follower2.getTopBundlesLoadDataStore().pushAsync(bundle, topBundlesExpected)
                 .get(3, TimeUnit.SECONDS);
+
+        // Confirm a stable channel owner before yielding to the next test's @BeforeMethod.
+        awaitChannelOwnerStable();
     }
 
     @Test(priority = Integer.MIN_VALUE)


(pulsar) 03/05: [fix][test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership (#25977)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 476881fab2da836aa0d593aa679f489a928f8a73
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Jun 8 23:41:13 2026 +0300

    [fix][test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership (#25977)
---
 .../ExtensibleLoadManagerImplBaseTest.java         | 77 +++++++++++++++++-----
 .../extensions/ExtensibleLoadManagerImplTest.java  |  3 +-
 2 files changed, 64 insertions(+), 16 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
index ea74bd86127..afd8681e24d 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
@@ -20,6 +20,7 @@ package org.apache.pulsar.broker.loadbalance.extensions;
 
 import static org.mockito.Mockito.reset;
 import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertTrue;
 import com.google.common.collect.Sets;
 import com.google.common.io.Resources;
 import java.util.ArrayList;
@@ -38,7 +39,6 @@ import org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateM
 import org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateTableViewImpl;
 import org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder;
 import org.apache.pulsar.broker.testcontext.PulsarTestContext;
-import org.apache.pulsar.client.admin.PulsarAdminException;
 import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException;
 import org.apache.pulsar.client.impl.LookupService;
@@ -48,6 +48,7 @@ import org.apache.pulsar.common.naming.TopicName;
 import org.apache.pulsar.common.policies.data.ClusterData;
 import org.apache.pulsar.common.policies.data.TenantInfoImpl;
 import org.awaitility.Awaitility;
+import org.awaitility.core.ConditionTimeoutException;
 import org.testng.annotations.AfterClass;
 import org.testng.annotations.BeforeClass;
 import org.testng.annotations.BeforeMethod;
@@ -196,18 +197,39 @@ public abstract class ExtensibleLoadManagerImplBaseTest extends MockedPulsarServ
     }
 
     @BeforeMethod(alwaysRun = true)
-    protected void initializeState() throws PulsarAdminException, IllegalAccessException {
-        // After a prior test churned leader election, the channel-topic bundle can be left
-        // unserved ("not served by this instance"), making the unload's channel publish fail
-        // (HTTP 500) or hang server-side. monitor() only self-heals when there is *no* channel
-        // owner; it does NOT heal the case where an owner is recorded but the bundle is not
-        // actually served, so the unload below can never publish. Force-serve the channel topic
-        // each attempt: an admin lookup re-assigns the pulsar/system bundle and getStats makes
-        // the owner load the topic (the lookup layer alone can claim an owner that refuses to
-        // serve). Bound each unload attempt and fail loudly on exhaustion.
+    protected void initializeState() throws Exception {
+        // Reset to a clean state before each test: reconcile each broker's role with the channel
+        // ownership and unload the test namespace so no bundle ownership carries over. The unload
+        // publishes a state change on the channel system topic.
+        //
+        // A prior role-churning test (e.g. the direct playLeader()/playFollower() calls in
+        // testRoleChangeIdempotency) can leave the channel system topic owned by a broker that no
+        // longer serves it ("not served by this instance, redo the lookup"), with the channel
+        // producer stuck in escalating reconnect backoff, so the unload's channel publish keeps
+        // failing. Each unload attempt force-serves the channel topic (an admin lookup re-assigns
+        // the pulsar/system bundle and getStats makes the owner load it); if that still does not
+        // recover, force a clean channel owner via leader re-election (which reassigns and
+        // re-serves the channel topic and makes clients redo their lookups) and retry.
+        try {
+            awaitTestNamespaceUnloaded(30);
+        } catch (ConditionTimeoutException channelWedged) {
+            recoverChannelOwnership();
+            awaitTestNamespaceUnloaded(60);
+        }
+        reset(primaryLoadManager, secondaryLoadManager);
+        FieldUtils.writeDeclaredField(pulsarClient, "lookup", lookupService, true);
+        pulsar1.getConfig().setLoadBalancerMultiPhaseBundleUnload(true);
+        pulsar2.getConfig().setLoadBalancerMultiPhaseBundleUnload(true);
+    }
+
+    // Drive monitor() to reconcile roles and force-serve the channel topic (monitor() only
+    // self-heals when there is *no* channel owner, not when an owner is recorded but the bundle is
+    // not served), then unload. ignoreExceptions() retries transient channel-publish failures; each
+    // unload attempt is bounded so a synchronous unload cannot block longer than the retry window.
+    private void awaitTestNamespaceUnloaded(long atMostSeconds) {
         boolean systemTopicChannel =
                 serviceUnitStateTableViewClassName.equals(ServiceUnitStateTableViewImpl.class.getName());
-        Awaitility.await().atMost(120, TimeUnit.SECONDS)
+        Awaitility.await().atMost(atMostSeconds, TimeUnit.SECONDS)
                 .pollInterval(1, TimeUnit.SECONDS)
                 .ignoreExceptions()
                 .untilAsserted(() -> {
@@ -219,10 +241,35 @@ public abstract class ExtensibleLoadManagerImplBaseTest extends MockedPulsarServ
                     }
                     admin.namespaces().unloadAsync(defaultTestNamespace).get(15, TimeUnit.SECONDS);
                 });
-        reset(primaryLoadManager, secondaryLoadManager);
-        FieldUtils.writeDeclaredField(pulsarClient, "lookup", lookupService, true);
-        pulsar1.getConfig().setLoadBalancerMultiPhaseBundleUnload(true);
-        pulsar2.getConfig().setLoadBalancerMultiPhaseBundleUnload(true);
+    }
+
+    /**
+     * Force a clean channel owner via leader re-election. After heavy direct
+     * playLeader()/playFollower() churn the channel system topic can be left owned by a broker
+     * that no longer serves it, leaving the channel producer stuck on a stale lookup in
+     * escalating reconnect backoff. Closing the current owner's LeaderElectionService moves
+     * ownership to the other broker; its playLeader() re-creates and re-serves the channel
+     * topic, and the ownership change makes clients redo their (stale) lookups.
+     */
+    private void recoverChannelOwnership() throws Exception {
+        boolean pulsar1Owns;
+        try {
+            pulsar1Owns = channel1.isChannelOwner();
+        } catch (Exception e) {
+            // Owner can't be determined (e.g. no channel owner now); default to moving to pulsar2.
+            pulsar1Owns = true;
+        }
+        PulsarService currentOwner = pulsar1Owns ? pulsar1 : pulsar2;
+        ServiceUnitStateChannelImpl newOwnerChannel = pulsar1Owns ? channel2 : channel1;
+        currentOwner.getLeaderElectionService().close();
+        try {
+            Awaitility.await().atMost(30, TimeUnit.SECONDS).ignoreExceptions()
+                    .untilAsserted(() -> assertTrue(newOwnerChannel.isChannelOwner()));
+        } catch (ConditionTimeoutException ignore) {
+            // Best effort: the subsequent unload retry is the real backstop.
+        } finally {
+            currentOwner.getLeaderElectionService().start();
+        }
     }
 
     protected void setPrimaryLoadManager() throws IllegalAccessException {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
index 05e9bfba6ef..24a62650125 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplTest.java
@@ -1578,7 +1578,8 @@ public class ExtensibleLoadManagerImplTest extends ExtensibleLoadManagerImplBase
     // and trip TestNG's ThreadTimeoutException mid-poll, and any failure to settle
     // is swallowed-and-logged rather than thrown. That matters because callers invoke this from
     // a finally block — a settling delay here must never replace (mask) the body's exception.
-    // The next test's initializeState() carries a 60s ignoreExceptions retry as the real backstop.
+    // The next test's initializeState() is the real backstop: it retries the unload and, if the
+    // channel stays wedged, forces a clean channel owner via leader re-election before retrying.
     private void awaitChannelOwnerStable() {
         try {
             Awaitility.await().atMost(20, TimeUnit.SECONDS).ignoreExceptions().untilAsserted(() -> {


(pulsar) 02/05: [fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState (#25976)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 136bc884c213012aa355a694688aea86cd04ed8e
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Mon Jun 8 12:32:44 2026 -0700

    [fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState (#25976)
---
 .../extensions/ExtensibleLoadManagerImplBaseTest.java    | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
index 2b30723f0a2..ea74bd86127 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
@@ -199,16 +199,24 @@ public abstract class ExtensibleLoadManagerImplBaseTest extends MockedPulsarServ
     protected void initializeState() throws PulsarAdminException, IllegalAccessException {
         // After a prior test churned leader election, the channel-topic bundle can be left
         // unserved ("not served by this instance"), making the unload's channel publish fail
-        // (HTTP 500) or hang server-side until the background monitor task (120s interval)
-        // reconciles the brokers' roles with the channel ownership. Drive monitor() eagerly to
-        // heal that state, bound each unload attempt (a synchronous unload() can block longer
-        // than the whole retry window), and fail loudly on exhaustion.
+        // (HTTP 500) or hang server-side. monitor() only self-heals when there is *no* channel
+        // owner; it does NOT heal the case where an owner is recorded but the bundle is not
+        // actually served, so the unload below can never publish. Force-serve the channel topic
+        // each attempt: an admin lookup re-assigns the pulsar/system bundle and getStats makes
+        // the owner load the topic (the lookup layer alone can claim an owner that refuses to
+        // serve). Bound each unload attempt and fail loudly on exhaustion.
+        boolean systemTopicChannel =
+                serviceUnitStateTableViewClassName.equals(ServiceUnitStateTableViewImpl.class.getName());
         Awaitility.await().atMost(120, TimeUnit.SECONDS)
                 .pollInterval(1, TimeUnit.SECONDS)
                 .ignoreExceptions()
                 .untilAsserted(() -> {
                     primaryLoadManager.monitor();
                     secondaryLoadManager.monitor();
+                    if (systemTopicChannel) {
+                        admin.lookups().lookupTopic(ServiceUnitStateTableViewImpl.TOPIC);
+                        admin.topics().getStats(ServiceUnitStateTableViewImpl.TOPIC);
+                    }
                     admin.namespaces().unloadAsync(defaultTestNamespace).get(15, TimeUnit.SECONDS);
                 });
         reset(primaryLoadManager, secondaryLoadManager);