You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2026/09/14 09:56:29 UTC

(pulsar) branch branch-4.2 updated (e3a60eac6029 -> 070583eaa929)

This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git


    from e3a60eac6029 [fix][broker] Fix multi-role authorization regressions and optimize nested checks (#26551)
     new 63b4721a62ad [fix][broker] Send ActiveConsumerChange for non-persistent Failover subscriptions (#26482)
     new 3eb8ae0a3f53 [improve][broker] Make exposing topic level metric dynamic (#21006)
     new 73d3695880ce [fix][admin] Allow defaultNumPartitions for non-partitioned autoTopicCreation override (#25163)
     new 3d09c6b3f9d6 [fix][common] Handle synchronous failures from future suppliers (#25939)
     new 8f43058b25aa [fix][fn] Bound function package downloads from BookKeeper and stop SchedulerManager.close() from hanging on a stuck scheduling round (#26556)
     new 89184b53edb9 [fix][client] Keep transactional and non-transactional messages in separate batches (#26547)
     new e9954e1506f1 [fix][test] Preserve class loading in broker interceptor fixtures (#26567)
     new e95cbfde259d [fix][test] Await both DLQ producers in multi-consumer test (#26561)
     new 775373cebf27 [fix][test] Allow for CLI startup in function integration tests (#26564)
     new d03b88faf8e6 [fix][test] Isolate mutable broker fixtures between test methods (#26563)
     new 078f7b2ab9dc [fix][client] Complete table view refresh after applying messages (#26566)
     new 2a05e1dcdba0 [fix][test] Cover scheduler shutdown membership-lock deadlock (#26557)
     new 2766ade8c41c [fix] Fix cursor deadlock when skipping non-recoverable entries during mark-delete persistence (#26570)
     new bc7447b02a3a [fix][txn] Stop the pending ack replay loop from spinning forever (#26369)
     new 070583eaa929 [fix][txn] Prevent queued snapshot writes from stalling (#26565)

The 15 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../bookkeeper/mledger/impl/ManagedCursorImpl.java |  37 +--
 .../apache/bookkeeper/mledger/util/Futures.java    |   2 +-
 .../bookkeeper/mledger/impl/ManagedCursorTest.java |  38 +++
 .../bookkeeper/mledger/util/FuturesTest.java       |  41 +--
 .../apache/pulsar/broker/ServiceConfiguration.java |  21 +-
 .../org/apache/pulsar/broker/PulsarService.java    |   6 +-
 ...onPersistentDispatcherSingleActiveConsumer.java |   4 +-
 .../prometheus/PrometheusMetricsGenerator.java     |  31 +-
 .../prometheus/PulsarPrometheusMetricsServlet.java |  13 +-
 .../SnapshotSegmentAbortedTxnProcessorImpl.java    |  13 +-
 .../pendingack/impl/MLPendingAckStore.java         | 150 +++++++--
 .../pendingack/impl/PendingAckHandleImpl.java      |  43 ++-
 .../pendingack/impl/PendingAckHandleState.java     |   7 +-
 .../apache/pulsar/broker/admin/AdminApi2Test.java  |  48 +++
 .../broker/intercept/BrokerInterceptorTest.java    |   7 +-
 .../intercept/ExceptionsBrokerInterceptorTest.java |   7 +-
 .../service/BrokerDispatchRateLimiterTest.java     |   8 +-
 .../BrokerServiceAutoTopicCreationTest.java        |  12 +-
 .../pulsar/broker/service/BrokerServiceTest.java   |  57 ++--
 .../SetReplicationClustersValidationTest.java      |   2 +-
 ...onPersistentDispatcherFailoverConsumerTest.java | 247 ++++++++++++++
 .../pulsar/broker/stats/PrometheusMetricsTest.java |  32 ++
 .../broker/transaction/TransactionProduceTest.java |  55 ++++
 ...apshotSegmentAbortedTxnProcessorWorkerTest.java | 124 +++++++
 .../pendingack/impl/MLPendingAckStoreTest.java     | 362 +++++++++++++++++++++
 .../pulsar/client/api/DeadLetterTopicTest.java     |   5 +-
 .../worker/WorkerUtilsBookKeeperPackageTest.java   | 144 ++++++++
 .../data/impl/AutoTopicCreationOverrideImpl.java   |   8 +-
 .../client/impl/AbstractBatchMessageContainer.java |  22 +-
 .../client/impl/BatchMessageKeyBasedContainer.java |  13 +
 .../apache/pulsar/client/impl/TableViewImpl.java   |   8 +-
 .../client/impl/BatchMessageContainerImplTest.java | 151 +++++++++
 .../pulsar/client/impl/TableViewImplTest.java      |  90 +++++
 .../org/apache/pulsar/common/util/FutureUtil.java  |  56 +++-
 .../data/AutoTopicCreationOverrideTest.java        |  24 +-
 .../apache/pulsar/common/util/FutureUtilTest.java  |  89 ++++-
 pulsar-functions/worker/pom.xml                    |   6 +
 .../pulsar/functions/worker/SchedulerManager.java  |  78 ++++-
 .../pulsar/functions/worker/WorkerUtils.java       |   4 +-
 .../functions/worker/dlog/DLInputStream.java       | 152 ++++++---
 .../functions/worker/SchedulerManagerTest.java     |  95 ++++++
 .../functions/worker/dlog/DLInputStreamTest.java   | 149 +++++++--
 .../integration/functions/PulsarFunctionsTest.java |   3 +-
 .../integration/functions/PulsarStateTest.java     |   6 +-
 44 files changed, 2183 insertions(+), 287 deletions(-)
 copy tiered-storage/jcloud/src/test/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedInputStreamTest.java => managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/FuturesTest.java (50%)
 create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherFailoverConsumerTest.java
 create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorWorkerTest.java
 create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/WorkerUtilsBookKeeperPackageTest.java


(pulsar) 02/15: [improve][broker] Make exposing topic level metric dynamic (#21006)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 3eb8ae0a3f53e58f76eabd1b44a0546fcae5c205
Author: Hang Chen <ch...@apache.org>
AuthorDate: Sun Sep 13 03:06:46 2026 +0800

    [improve][broker] Make exposing topic level metric dynamic (#21006)
    
    Co-authored-by: Lari Hotari <lh...@apache.org>
    (cherry picked from commit 91f690ff7b30a5d4160a3eb80ab0cc709c70195f)
---
 .../apache/pulsar/broker/ServiceConfiguration.java | 21 +++++++++-----
 .../org/apache/pulsar/broker/PulsarService.java    |  6 +---
 .../prometheus/PrometheusMetricsGenerator.java     | 31 ++++++++++++++++-----
 .../prometheus/PulsarPrometheusMetricsServlet.java | 13 +++++++--
 .../pulsar/broker/stats/PrometheusMetricsTest.java | 32 ++++++++++++++++++++++
 5 files changed, 81 insertions(+), 22 deletions(-)

diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 664c7ad6e29f..ccf62c04b4b2 100644
--- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -3531,9 +3531,10 @@ public class ServiceConfiguration implements PulsarConfiguration {
     private boolean authenticateMetricsEndpoint = false;
     @FieldContext(
         category = CATEGORY_METRICS,
+        dynamic = true,
         doc = "If true, export topic level metrics otherwise namespace level"
     )
-    private boolean exposeTopicLevelMetricsInPrometheus = true;
+    private volatile boolean exposeTopicLevelMetricsInPrometheus = true;
     @FieldContext(
             category = CATEGORY_METRICS,
             doc = "Set to true to enable the broker to cache the metrics response; the default is false. "
@@ -3543,24 +3544,28 @@ public class ServiceConfiguration implements PulsarConfiguration {
     private boolean metricsBufferResponse = false;
     @FieldContext(
         category = CATEGORY_METRICS,
+        dynamic = true,
         doc = "If true, export consumer level metrics otherwise namespace level"
     )
-    private boolean exposeConsumerLevelMetricsInPrometheus = false;
+    private volatile boolean exposeConsumerLevelMetricsInPrometheus = false;
     @FieldContext(
             category = CATEGORY_METRICS,
+            dynamic = true,
             doc = "If true, export producer level metrics otherwise namespace level"
     )
-    private boolean exposeProducerLevelMetricsInPrometheus = false;
+    private volatile boolean exposeProducerLevelMetricsInPrometheus = false;
     @FieldContext(
             category = CATEGORY_METRICS,
+            dynamic = true,
             doc = "If true, export managed ledger metrics (aggregated by namespace)"
     )
-    private boolean exposeManagedLedgerMetricsInPrometheus = true;
+    private volatile boolean exposeManagedLedgerMetricsInPrometheus = true;
     @FieldContext(
             category = CATEGORY_METRICS,
+            dynamic = true,
             doc = "If true, export managed cursor metrics"
     )
-    private boolean exposeManagedCursorMetricsInPrometheus = false;
+    private volatile boolean exposeManagedCursorMetricsInPrometheus = false;
     @FieldContext(
             category = CATEGORY_METRICS,
             doc = "Classname of Pluggable JVM GC metrics logger that can log GC specific metrics")
@@ -3568,11 +3573,12 @@ public class ServiceConfiguration implements PulsarConfiguration {
 
     @FieldContext(
         category = CATEGORY_METRICS,
+        dynamic = true,
         doc = "Enable expose the precise backlog stats.\n"
                 + " Set false to use published counter and consumed counter to calculate,\n"
                 + " this would be more efficient but may be inaccurate. Default is false."
     )
-    private boolean exposePreciseBacklogInPrometheus = false;
+    private volatile boolean exposePreciseBacklogInPrometheus = false;
 
     @FieldContext(
         category = CATEGORY_METRICS,
@@ -3584,10 +3590,11 @@ public class ServiceConfiguration implements PulsarConfiguration {
 
     @FieldContext(
             category = CATEGORY_METRICS,
+            dynamic = true,
             doc = "Enable expose the backlog size for each subscription when generating stats.\n"
                     + " Locking is used for fetching the status so default to false."
     )
-    private boolean exposeSubscriptionBacklogSizeInPrometheus = false;
+    private volatile boolean exposeSubscriptionBacklogSizeInPrometheus = false;
 
     @FieldContext(
             category = CATEGORY_METRICS,
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
index 98ba2b6cf12e..5f2bb72049e9 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java
@@ -1183,11 +1183,7 @@ public class PulsarService implements AutoCloseable, ShutdownService {
     }
 
     private synchronized void createMetricsServlet() {
-        this.metricsServlet = new PulsarPrometheusMetricsServlet(
-                this, config.isExposeTopicLevelMetricsInPrometheus(),
-                config.isExposeConsumerLevelMetricsInPrometheus(),
-                config.isExposeProducerLevelMetricsInPrometheus(),
-                config.isSplitTopicAndPartitionLabelInPrometheus());
+        this.metricsServlet = new PulsarPrometheusMetricsServlet(this);
         if (pendingMetricsProviders != null) {
             pendingMetricsProviders.forEach(provider -> metricsServlet.addRawMetricsProvider(provider));
             this.pendingMetricsProviders = null;
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
index 2bb6372c1cbf..285b6aaefc93 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
@@ -47,6 +47,7 @@ import java.util.concurrent.Executor;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.BooleanSupplier;
 import java.util.zip.CRC32;
 import java.util.zip.Deflater;
 import lombok.extern.slf4j.Slf4j;
@@ -297,10 +298,10 @@ public class PrometheusMetricsGenerator implements AutoCloseable {
     }
 
     private final PulsarService pulsar;
-    private final boolean includeTopicMetrics;
-    private final boolean includeConsumerMetrics;
-    private final boolean includeProducerMetrics;
-    private final boolean splitTopicAndPartitionIndexLabel;
+    private final BooleanSupplier includeTopicMetrics;
+    private final BooleanSupplier includeConsumerMetrics;
+    private final BooleanSupplier includeProducerMetrics;
+    private final BooleanSupplier splitTopicAndPartitionIndexLabel;
     private final Clock clock;
 
     private volatile int initialBufferSize = DEFAULT_INITIAL_BUFFER_SIZE;
@@ -308,6 +309,20 @@ public class PrometheusMetricsGenerator implements AutoCloseable {
     public PrometheusMetricsGenerator(PulsarService pulsar, boolean includeTopicMetrics,
                                       boolean includeConsumerMetrics, boolean includeProducerMetrics,
                                       boolean splitTopicAndPartitionIndexLabel, Clock clock) {
+        this(pulsar, () -> includeTopicMetrics, () -> includeConsumerMetrics, () -> includeProducerMetrics,
+                () -> splitTopicAndPartitionIndexLabel, clock);
+    }
+
+    public PrometheusMetricsGenerator(PulsarService pulsar, Clock clock) {
+        this(pulsar, pulsar.getConfiguration()::isExposeTopicLevelMetricsInPrometheus,
+                pulsar.getConfiguration()::isExposeConsumerLevelMetricsInPrometheus,
+                pulsar.getConfiguration()::isExposeProducerLevelMetricsInPrometheus,
+                pulsar.getConfiguration()::isSplitTopicAndPartitionLabelInPrometheus, clock);
+    }
+
+    private PrometheusMetricsGenerator(PulsarService pulsar, BooleanSupplier includeTopicMetrics,
+                                       BooleanSupplier includeConsumerMetrics, BooleanSupplier includeProducerMetrics,
+                                       BooleanSupplier splitTopicAndPartitionIndexLabel, Clock clock) {
         this.pulsar = pulsar;
         this.includeTopicMetrics = includeTopicMetrics;
         this.includeConsumerMetrics = includeConsumerMetrics;
@@ -326,15 +341,17 @@ public class PrometheusMetricsGenerator implements AutoCloseable {
 
             generateSystemMetrics(stream, pulsar.getConfiguration().getClusterName());
 
-            NamespaceStatsAggregator.generate(pulsar, includeTopicMetrics, includeConsumerMetrics,
-                    includeProducerMetrics, splitTopicAndPartitionIndexLabel, metricStreams);
+            boolean exportTopicMetrics = includeTopicMetrics.getAsBoolean();
+            NamespaceStatsAggregator.generate(pulsar, exportTopicMetrics, includeConsumerMetrics.getAsBoolean(),
+                    includeProducerMetrics.getAsBoolean(), splitTopicAndPartitionIndexLabel.getAsBoolean(),
+                    metricStreams);
 
             if (pulsar.getWorkerServiceOpt().isPresent()) {
                 pulsar.getWorkerService().generateFunctionsStats(stream);
             }
 
             if (pulsar.getConfiguration().isTransactionCoordinatorEnabled()) {
-                TransactionAggregator.generate(pulsar, metricStreams, includeTopicMetrics);
+                TransactionAggregator.generate(pulsar, metricStreams, exportTopicMetrics);
             }
 
             metricStreams.flushAllToStream(stream);
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PulsarPrometheusMetricsServlet.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PulsarPrometheusMetricsServlet.java
index 7593c6697259..5260249a22bb 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PulsarPrometheusMetricsServlet.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PulsarPrometheusMetricsServlet.java
@@ -48,12 +48,19 @@ public class PulsarPrometheusMetricsServlet extends PrometheusMetricsServlet {
     public PulsarPrometheusMetricsServlet(PulsarService pulsar, boolean includeTopicMetrics,
                                           boolean includeConsumerMetrics, boolean includeProducerMetrics,
                                           boolean splitTopicAndPartitionLabel) {
+        this(pulsar, new PrometheusMetricsGenerator(pulsar, includeTopicMetrics, includeConsumerMetrics,
+                includeProducerMetrics, splitTopicAndPartitionLabel, Clock.systemUTC()));
+    }
+
+    public PulsarPrometheusMetricsServlet(PulsarService pulsar) {
+        this(pulsar, new PrometheusMetricsGenerator(pulsar, Clock.systemUTC()));
+    }
+
+    private PulsarPrometheusMetricsServlet(PulsarService pulsar, PrometheusMetricsGenerator metricsGenerator) {
         super(pulsar.getConfiguration().getMetricsServletTimeoutMs(), pulsar.getConfiguration().getClusterName(),
                 EXECUTOR_MAX_THREADS);
         MetricsExports.initialize();
-        prometheusMetricsGenerator =
-                new PrometheusMetricsGenerator(pulsar, includeTopicMetrics, includeConsumerMetrics,
-                        includeProducerMetrics, splitTopicAndPartitionLabel, Clock.systemUTC());
+        prometheusMetricsGenerator = metricsGenerator;
         gzipCompressionEnabledForMetrics = isGzipCompressionEnabledForEndpoint(
                 pulsar.getConfiguration().getHttpServerGzipCompressionExcludedPaths(), DEFAULT_METRICS_PATH);
     }
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
index 2df7f2b973a7..4a248935a387 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java
@@ -90,6 +90,7 @@ import org.apache.pulsar.broker.service.Topic;
 import org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor;
 import org.apache.pulsar.broker.service.persistent.PersistentSubscription;
 import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient;
 import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator;
 import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.Message;
@@ -139,6 +140,37 @@ public class PrometheusMetricsTest extends BrokerTestBase {
         super.internalCleanup();
     }
 
+    @Test
+    public void testDynamicMetricsConfiguration() throws Exception {
+        String topic = "persistent://my-property/my-ns/dynamic-metrics";
+        try (Producer<byte[]> producer = pulsarClient.newProducer().topic(topic)
+                     .producerName("dynamic-producer").create();
+             Consumer<byte[]> consumer = pulsarClient.newConsumer().topic(topic)
+                     .subscriptionName("dynamic-subscription").consumerName("dynamic-consumer").subscribe()) {
+            producer.send(new byte[] {1});
+            consumer.acknowledge(consumer.receive(10, TimeUnit.SECONDS));
+            PrometheusMetricsClient metricsClient = new PrometheusMetricsClient("localhost",
+                    pulsar.getListenPortHTTP().orElseThrow());
+            for (boolean enabled : new boolean[] {false, true, false}) {
+                admin.brokers().updateDynamicConfiguration("exposeTopicLevelMetricsInPrometheus",
+                        Boolean.toString(enabled));
+                admin.brokers().updateDynamicConfiguration("exposeConsumerLevelMetricsInPrometheus",
+                        Boolean.toString(enabled));
+                admin.brokers().updateDynamicConfiguration("exposeProducerLevelMetricsInPrometheus",
+                        Boolean.toString(enabled));
+                Awaitility.await().untilAsserted(() -> {
+                    Collection<Metric> metrics = metricsClient.getMetrics().getNameToDataPoints().values();
+                    assertThat(metrics.stream().anyMatch(metric -> topic.equals(metric.tags.get("topic"))))
+                            .isEqualTo(enabled);
+                    assertThat(metrics.stream().anyMatch(metric ->
+                            "dynamic-producer".equals(metric.tags.get("producer_name")))).isEqualTo(enabled);
+                    assertThat(metrics.stream().anyMatch(metric ->
+                            "dynamic-consumer".equals(metric.tags.get("consumer_name")))).isEqualTo(enabled);
+                });
+            }
+        }
+    }
+
     @Test
     public void testPublishRateLimitedTimes() throws Exception {
         cleanup();


(pulsar) 06/15: [fix][client] Keep transactional and non-transactional messages in separate batches (#26547)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 89184b53edb9ae9d5fbde66c5ed41528e1ed4aac
Author: SongOf <46...@users.noreply.github.com>
AuthorDate: Sun Sep 13 04:46:37 2026 +0800

    [fix][client] Keep transactional and non-transactional messages in separate batches (#26547)
    
    Co-authored-by: maxlisongsong <ma...@didiglobal.com>
    (cherry picked from commit fa05c3e71d22dc4f4fd69a09385a6c0c5ae5f844)
---
 .../broker/transaction/TransactionProduceTest.java |  55 ++++++++
 .../client/impl/AbstractBatchMessageContainer.java |  22 ++-
 .../client/impl/BatchMessageKeyBasedContainer.java |  13 ++
 .../client/impl/BatchMessageContainerImplTest.java | 151 +++++++++++++++++++++
 4 files changed, 237 insertions(+), 4 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionProduceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionProduceTest.java
index 757f86ec834d..5dc71db4d732 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionProduceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionProduceTest.java
@@ -51,6 +51,7 @@ import org.apache.pulsar.client.api.MessageId;
 import org.apache.pulsar.client.api.Producer;
 import org.apache.pulsar.client.api.PulsarClient;
 import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.client.api.SubscriptionInitialPosition;
 import org.apache.pulsar.client.api.SubscriptionType;
 import org.apache.pulsar.client.api.transaction.Transaction;
@@ -92,6 +93,60 @@ public class TransactionProduceTest extends TransactionTestBase {
         super.internalCleanup();
     }
 
+    /**
+     * A batch carries one transaction id in its metadata, so every message in it inherits that transaction.
+     * A plain message batched together with transactional ones is therefore discarded when that transaction
+     * aborts, even though the application never sent it inside a transaction.
+     */
+    @Test
+    public void testAbortedTransactionDoesNotDiscardPlainMessageBatchedWithIt() throws Exception {
+        final String topic = NAMESPACE1 + "/txn-batch-isolation";
+        admin.topics().createNonPartitionedTopic(topic);
+
+        @Cleanup
+        Consumer<String> consumer = pulsarClient.newConsumer(Schema.STRING)
+                .topic(topic)
+                .subscriptionName("txn-batch-isolation-sub")
+                .subscribe();
+
+        Transaction txn = pulsarClient.newTransaction()
+                .withTransactionTimeout(60, TimeUnit.SECONDS)
+                .build().get();
+
+        @Cleanup
+        Producer<String> producer = pulsarClient.newProducer(Schema.STRING)
+                .topic(topic)
+                .sendTimeout(0, TimeUnit.SECONDS)
+                .enableBatching(true)
+                .batchingMaxMessages(100)
+                // Long enough that the two sends under test cannot be split into separate batches by the
+                // timer, short enough that a batch still closes on its own when a flush cannot reach it.
+                .batchingMaxPublishDelay(5, TimeUnit.SECONDS)
+                .create();
+
+        // The first transactional send on a topic waits for the topic to be registered with the transaction
+        // coordinator before the message reaches the batch. Get that round trip out of the way, so the two
+        // sends under test are added to the container synchronously and therefore land in one batch.
+        // A flush issued here would run before the message reaches the batch, so let the batch timer close it.
+        producer.newMessage(txn).value("warm-up").sendAsync().get(30, TimeUnit.SECONDS);
+
+        producer.newMessage(txn).value("in-txn").sendAsync();
+        CompletableFuture<MessageId> plainSend = producer.newMessage().value("plain").sendAsync();
+        producer.flush();
+        plainSend.get(30, TimeUnit.SECONDS);
+
+        // The topic's max read position does not advance past an ongoing transaction, so nothing on this topic
+        // is readable until the transaction ends. Abort it: that discards the transactional messages, and the
+        // plain one must survive because it was never part of the transaction.
+        txn.abort().get(60, TimeUnit.SECONDS);
+
+        Message<String> received = consumer.receive(30, TimeUnit.SECONDS);
+        Assert.assertNotNull(received, "the plain message was discarded by the aborted transaction");
+        Assert.assertEquals(received.getValue(), "plain");
+
+        // Both transactional messages were aborted, so nothing else may arrive.
+        Assert.assertNull(consumer.receive(3, TimeUnit.SECONDS));
+    }
 
     @Test
     public void produceAndCommitTest() throws Exception {
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AbstractBatchMessageContainer.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AbstractBatchMessageContainer.java
index 3ba7866350a0..9f068dbb0969 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AbstractBatchMessageContainer.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AbstractBatchMessageContainer.java
@@ -120,14 +120,28 @@ public abstract class AbstractBatchMessageContainer implements BatchMessageConta
         this.maxBytesInBatch = producer.getConfiguration().getBatchingMaxBytes();
     }
 
+    /**
+     * Whether {@code msg} belongs to the same transaction as the messages already in this batch.
+     *
+     * <p>A batch carries a single transaction id in its metadata, so every message in it inherits that
+     * transaction. "No transaction" is therefore an identity of its own and is not compatible with any
+     * transaction: mixing the two in one batch would either enroll a plain message in a transaction (invisible
+     * until commit, dropped on abort) or publish a transactional message outside its transaction.
+     *
+     * <p>This is a pure query. The batch adopts its transaction id from its first message in {@code add}.
+     */
     @Override
     public boolean hasSameTxn(MessageImpl<?> msg) {
-        if (!msg.getMessageBuilder().hasTxnidMostBits() || !msg.getMessageBuilder().hasTxnidLeastBits()) {
+        if (numMessagesInBatch == 0) {
             return true;
         }
-        if (currentTxnidMostBits == -1 || currentTxnidLeastBits == -1) {
-            currentTxnidMostBits = msg.getMessageBuilder().getTxnidMostBits();
-            currentTxnidLeastBits = msg.getMessageBuilder().getTxnidLeastBits();
+        boolean msgHasTxn = msg.getMessageBuilder().hasTxnidMostBits()
+                && msg.getMessageBuilder().hasTxnidLeastBits();
+        boolean batchHasTxn = currentTxnidMostBits != -1L && currentTxnidLeastBits != -1L;
+        if (msgHasTxn != batchHasTxn) {
+            return false;
+        }
+        if (!msgHasTxn) {
             return true;
         }
         return currentTxnidMostBits == msg.getMessageBuilder().getTxnidMostBits()
diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageKeyBasedContainer.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageKeyBasedContainer.java
index f6e3f5a683ed..96074b43466f 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageKeyBasedContainer.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageKeyBasedContainer.java
@@ -46,6 +46,19 @@ class BatchMessageKeyBasedContainer extends AbstractBatchMessageContainer {
             log.debug("[{}] [{}] add message to batch, num messages in batch so far is {}", topicName,
                     producer.getProducerName(), numMessagesInBatch);
         }
+        if (numMessagesInBatch == 0) {
+            // The whole container shares one transaction identity; hasSameTxn keeps later messages consistent
+            // with it. Capturing it here rather than in hasSameTxn keeps that check side-effect free.
+            //
+            // Both fields are assigned unconditionally, so a plain first message resets them. An inner batch
+            // whose first add fails to allocate clears itself and stays empty, which leaves the count at zero
+            // without clearing this container: a leftover transaction id would then be inherited by the next,
+            // plain, first message and would wrongly admit later messages of that transaction.
+            boolean msgHasTxn = msg.getMessageBuilder().hasTxnidMostBits()
+                    && msg.getMessageBuilder().hasTxnidLeastBits();
+            currentTxnidMostBits = msgHasTxn ? msg.getMessageBuilder().getTxnidMostBits() : -1L;
+            currentTxnidLeastBits = msgHasTxn ? msg.getMessageBuilder().getTxnidLeastBits() : -1L;
+        }
         String key = getKey(msg);
         final BatchMessageContainerImpl batchMessageContainer = batches.computeIfAbsent(key,
                 __ -> new BatchMessageContainerImpl(producer));
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/BatchMessageContainerImplTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/BatchMessageContainerImplTest.java
index 3b2ac54b4bda..4b1aa23ac4f9 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/BatchMessageContainerImplTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/BatchMessageContainerImplTest.java
@@ -20,10 +20,13 @@ package org.apache.pulsar.client.impl;
 
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyLong;
 import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 import io.netty.buffer.ByteBufAllocator;
@@ -199,4 +202,152 @@ public class BatchMessageContainerImplTest {
         batchMessageContainer.clear();
         messages.forEach(ReferenceCountUtil::safeRelease);
     }
+
+    private ProducerImpl<?> createTestProducer() throws Exception {
+        ProducerImpl<?> producer = mock(ProducerImpl.class);
+        ProducerConfigurationData producerConfigurationData = new ProducerConfigurationData();
+        producerConfigurationData.setCompressionType(CompressionType.NONE);
+        PulsarClientImpl pulsarClient = mock(PulsarClientImpl.class);
+        when(pulsarClient.getCnxPool()).thenReturn(mock(ConnectionPool.class));
+        when(pulsarClient.getMemoryLimitController()).thenReturn(mock(MemoryLimitController.class));
+        Field clientField = HandlerState.class.getDeclaredField("client");
+        clientField.setAccessible(true);
+        clientField.set(producer, pulsarClient);
+        when(producer.getConfiguration()).thenReturn(producerConfigurationData);
+        when(producer.encryptMessage(any(), any())).thenAnswer(__ -> ByteBufAllocator.DEFAULT.buffer()
+                .writeBytes("payload".getBytes(StandardCharsets.UTF_8)));
+        return producer;
+    }
+
+    /**
+     * A batch carries one transaction id in its metadata, so every message in it inherits that transaction.
+     * Mixing a plain message into a transactional batch therefore silently enrolls it in the transaction: it
+     * stays invisible until commit and is dropped on abort, even though the application never sent it inside
+     * a transaction.
+     */
+    @Test
+    public void testPlainMessageIsNotBatchedWithTransactionalMessages() throws Exception {
+        BatchMessageContainerImpl container = new BatchMessageContainerImpl(createTestProducer());
+        try {
+            container.add(newMessage(1L, 7L, 13L), null);
+
+            assertFalse(container.hasSameTxn(newMessage(2L, null, null)),
+                    "a plain message was accepted into a transactional batch");
+        } finally {
+            container.discard(null);
+        }
+    }
+
+    /**
+     * The mirror image: a transactional message joining a batch that already holds plain messages stamps its
+     * transaction id onto the whole batch, retroactively pulling those plain messages into the transaction.
+     */
+    @Test
+    public void testTransactionalMessageIsNotBatchedWithPlainMessages() throws Exception {
+        BatchMessageContainerImpl container = new BatchMessageContainerImpl(createTestProducer());
+        try {
+            container.add(newMessage(1L, null, null), null);
+
+            assertFalse(container.hasSameTxn(newMessage(2L, 7L, 13L)),
+                    "a transactional message was accepted into a plain batch");
+        } finally {
+            container.discard(null);
+        }
+    }
+
+    /**
+     * Key-based batching fails the other way round: the guard reads the outer container's transaction state
+     * while the batch metadata is built by the per-key inner container, so a transactional message landing in
+     * a key bucket whose first message was plain is published with no transaction id at all — outside the
+     * transaction, and not rolled back on abort.
+     */
+    @Test
+    public void testKeyBasedContainerKeepsTransactionalAndPlainMessagesApart() throws Exception {
+        BatchMessageKeyBasedContainer container = new BatchMessageKeyBasedContainer();
+        container.setProducer(createTestProducer());
+        try {
+            container.add(newMessage(1L, null, null), null);
+
+            assertFalse(container.hasSameTxn(newMessage(2L, 7L, 13L)),
+                    "a transactional message was accepted into a plain key-based batch");
+        } finally {
+            container.discard(null);
+        }
+    }
+
+    /** Messages of the same transaction must still batch together, and two different transactions must not. */
+    @Test
+    public void testSameTransactionStillBatchesTogether() throws Exception {
+        BatchMessageContainerImpl container = new BatchMessageContainerImpl(createTestProducer());
+        try {
+            container.add(newMessage(1L, 7L, 13L), null);
+
+            assertTrue(container.hasSameTxn(newMessage(2L, 7L, 13L)),
+                    "two messages of the same transaction were split across batches");
+            assertFalse(container.hasSameTxn(newMessage(3L, 7L, 14L)),
+                    "messages of two different transactions were put in one batch");
+        } finally {
+            container.discard(null);
+        }
+    }
+
+    /** Plain messages must still batch with each other. */
+    @Test
+    public void testPlainMessagesStillBatchTogether() throws Exception {
+        BatchMessageContainerImpl container = new BatchMessageContainerImpl(createTestProducer());
+        try {
+            container.add(newMessage(1L, null, null), null);
+
+            assertTrue(container.hasSameTxn(newMessage(2L, null, null)),
+                    "two plain messages were split across batches");
+        } finally {
+            container.discard(null);
+        }
+    }
+
+    private static MessageImpl<byte[]> newMessage(long sequenceId, Long txnIdMostBits, Long txnIdLeastBits) {
+        MessageMetadata metadata = new MessageMetadata();
+        metadata.setSequenceId(sequenceId);
+        metadata.setProducerName("producer1");
+        metadata.setPublishTime(System.currentTimeMillis());
+        if (txnIdMostBits != null) {
+            metadata.setTxnidMostBits(txnIdMostBits);
+            metadata.setTxnidLeastBits(txnIdLeastBits);
+        }
+        ByteBuffer payload = ByteBuffer.wrap(("payload-" + sequenceId).getBytes(StandardCharsets.UTF_8));
+        return MessageImpl.create(metadata, payload, Schema.BYTES, null);
+    }
+
+    /**
+     * An inner batch whose first add fails to allocate clears itself and stays empty, so the outer container's
+     * message count is never incremented and the outer is not cleared either. Any transaction identity captured
+     * for that failed message must therefore not survive into the next first message, or a later message of that
+     * transaction would be admitted into a plain batch and published outside its transaction. Allocation failure
+     * is a recoverable path the client already handles.
+     */
+    @Test
+    public void testKeyBasedContainerDropsTheTransactionIdentityOfAFailedFirstAdd() throws Exception {
+        ProducerImpl<?> producer = createTestProducer();
+        MemoryLimitController memoryLimitController = producer.client.getMemoryLimitController();
+        // fail only the first inner-batch allocation
+        doThrow(new OutOfMemoryError("test")).doNothing()
+                .when(memoryLimitController).forceReserveMemory(anyLong());
+
+        BatchMessageKeyBasedContainer container = new BatchMessageKeyBasedContainer();
+        container.setProducer(producer);
+        try {
+            container.add(newMessage(1L, 7L, 13L), null);
+            assertEquals(container.getNumMessagesInBatch(), 0,
+                    "the failed add should have left the container empty");
+
+            container.add(newMessage(2L, null, null), null);
+
+            assertFalse(container.hasSameTxn(newMessage(3L, 7L, 13L)),
+                    "a transactional message was admitted into a plain batch through the transaction identity"
+                            + " left behind by a failed add");
+        } finally {
+            container.discard(null);
+        }
+    }
+
 }


(pulsar) 15/15: [fix][txn] Prevent queued snapshot writes from stalling (#26565)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 070583eaa9294e72f5a51dda2a2cc3ba577d3c1f
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Sep 14 03:49:07 2026 +0300

    [fix][txn] Prevent queued snapshot writes from stalling (#26565)
    
    (cherry picked from commit ce7c74f7a39372159805f9f1772215d6ed178dd4)
---
 .../SnapshotSegmentAbortedTxnProcessorImpl.java    |  13 ++-
 ...apshotSegmentAbortedTxnProcessorWorkerTest.java | 124 +++++++++++++++++++++
 2 files changed, 134 insertions(+), 3 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java
index eb31bed1a03f..71a666f351da 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java
@@ -610,6 +610,11 @@ public class SnapshotSegmentAbortedTxnProcessorImpl extends AbstractSnapshotAbor
             if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
                 //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
                 if (taskQueue.isEmpty()) {
+                    STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+                    // An append may have observed Operating between the empty check and releasing ownership.
+                    if (!taskQueue.isEmpty()) {
+                        executeTask();
+                    }
                     return;
                 }
                 Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
@@ -625,12 +630,14 @@ public class SnapshotSegmentAbortedTxnProcessorImpl extends AbstractSnapshotAbor
                     } else {
                         firstTask.getRight().getKey().complete(null);
                         taskQueue.removeFirst();
-                        //Execute the next task in the other thread.
-                        topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
-                                .getExecutor(this).submit(this::executeTask);
                     }
                     STATE_UPDATER.compareAndSet(this, OperationState.Operating,
                             OperationState.None);
+                    if (throwable == null) {
+                        // Release the operation before scheduling: the executor may run the next task immediately.
+                        topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+                                .getExecutor(this).submit(this::executeTask);
+                    }
                 });
             }
         }
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorWorkerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorWorkerTest.java
new file mode 100644
index 000000000000..08e43176c697
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorWorkerTest.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import com.google.common.util.concurrent.MoreExecutors;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.impl.SnapshotSegmentAbortedTxnProcessorImpl.PersistentWorker;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class SnapshotSegmentAbortedTxnProcessorWorkerTest {
+
+    private ExecutorService executor;
+    private SnapshotSegmentAbortedTxnProcessorImpl processor;
+    private PersistentWorker worker;
+
+    @BeforeMethod
+    public void setup() {
+        PersistentTopic topic = mock(PersistentTopic.class, RETURNS_DEEP_STUBS);
+        when(topic.getName()).thenReturn("persistent://public/default/snapshot-worker");
+        executor = MoreExecutors.newDirectExecutorService();
+        when(topic.getBrokerService().getPulsar().getTransactionExecutorProvider().getExecutor(any(Object.class)))
+                .thenReturn(executor);
+        processor = new SnapshotSegmentAbortedTxnProcessorImpl(topic);
+        worker = processor.new PersistentWorker(topic);
+    }
+
+    @Test(timeOut = 10_000)
+    public void testQueuedTasksContinueWhenExecutorRunsImmediately() throws Exception {
+        CompletableFuture<Void> firstWrite = new CompletableFuture<>();
+        CompletableFuture<Void> secondWrite = new CompletableFuture<>();
+        AtomicBoolean secondStarted = new AtomicBoolean();
+        AtomicBoolean thirdStarted = new AtomicBoolean();
+        CompletableFuture<Void> firstResult = worker.appendTask(PersistentWorker.OperationType.WriteSegment,
+                () -> firstWrite);
+        CompletableFuture<Void> secondResult = worker.appendTask(PersistentWorker.OperationType.WriteSegment, () -> {
+            secondStarted.set(true);
+            return secondWrite;
+        });
+        assertFalse(secondStarted.get());
+
+        firstWrite.complete(null);
+        firstResult.get(5, TimeUnit.SECONDS);
+        assertTrue(secondStarted.get(), "The queued write must start when the previous write completes");
+
+        CompletableFuture<Void> thirdResult = worker.appendTask(PersistentWorker.OperationType.WriteSegment, () -> {
+            thirdStarted.set(true);
+            return CompletableFuture.completedFuture(null);
+        });
+        assertFalse(thirdStarted.get(), "The next write must wait for the in-flight write");
+        secondWrite.complete(null);
+        secondResult.get(5, TimeUnit.SECONDS);
+        thirdResult.get(5, TimeUnit.SECONDS);
+        assertTrue(thirdStarted.get());
+    }
+
+    @Test(timeOut = 10_000)
+    public void testQueueDrainedBeforeAcquiringOperation() throws Exception {
+        AtomicInteger stage = new AtomicInteger();
+        AtomicReference<CompletableFuture<Void>> appendedWhileOperating = new AtomicReference<>();
+        worker.taskQueue = new ConcurrentLinkedDeque<>() {
+            @Override
+            public boolean isEmpty() {
+                boolean empty = super.isEmpty();
+                if (!empty && stage.compareAndSet(0, 1)) {
+                    // Another caller drains the queue after the initial snapshot, before ownership is acquired.
+                    worker.appendTask(PersistentWorker.OperationType.WriteSegment,
+                            () -> CompletableFuture.completedFuture(null));
+                    stage.set(2);
+                } else if (empty && stage.compareAndSet(2, 3)) {
+                    // An append observes Operating after the second empty snapshot, before ownership is released.
+                    appendedWhileOperating.set(worker.appendTask(PersistentWorker.OperationType.WriteSegment,
+                            () -> CompletableFuture.completedFuture(null)));
+                }
+                return empty;
+            }
+        };
+        worker.appendTask(PersistentWorker.OperationType.WriteSegment,
+                () -> CompletableFuture.completedFuture(null)).get(5, TimeUnit.SECONDS);
+        assertEquals(stage.get(), 3, "Both queue observations must be exercised");
+        appendedWhileOperating.get().get(5, TimeUnit.SECONDS);
+        worker.appendTask(PersistentWorker.OperationType.WriteSegment,
+                () -> CompletableFuture.completedFuture(null)).get(5, TimeUnit.SECONDS);
+    }
+
+    @AfterMethod(alwaysRun = true)
+    public void cleanup() throws Exception {
+        worker.closeAsync().get(5, TimeUnit.SECONDS);
+        processor.closeAsync().get(5, TimeUnit.SECONDS);
+        executor.shutdownNow();
+    }
+}


(pulsar) 14/15: [fix][txn] Stop the pending ack replay loop from spinning forever (#26369)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit bc7447b02a3ab6d0904e44be57f3c980dff22bd8
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Sep 14 03:48:00 2026 +0300

    [fix][txn] Stop the pending ack replay loop from spinning forever (#26369)
    
    (cherry picked from commit b28950ccfbb52ed20557ee3439d4bcceb125b525)
---
 .../pendingack/impl/MLPendingAckStore.java         | 150 +++++++--
 .../pendingack/impl/PendingAckHandleImpl.java      |  43 ++-
 .../pendingack/impl/PendingAckHandleState.java     |   7 +-
 .../pendingack/impl/MLPendingAckStoreTest.java     | 362 +++++++++++++++++++++
 4 files changed, 516 insertions(+), 46 deletions(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
index b4f63bf64c13..203306c95917 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java
@@ -407,44 +407,80 @@ public class MLPendingAckStore implements PendingAckStore {
                 while (lastConfirmedEntry.compareTo(currentLoadPosition) > 0 && fillEntryQueueCallback.fillQueue()) {
                     Entry entry = entryQueue.poll();
                     if (entry != null) {
-                        currentLoadPosition = PositionFactory.create(entry.getLedgerId(), entry.getEntryId());
-                        List<PendingAckMetadataEntry> logs = deserializeEntry(entry);
-                        if (logs.isEmpty()){
-                            continue;
-                        } else if (logs.size() == 1){
-                            currentIndexLag.incrementAndGet();
-                            PendingAckMetadataEntry log = logs.get(0);
-                            handleMetadataEntry(PositionFactory.create(entry.getLedgerId(), entry.getEntryId()), log);
-                            pendingAckReplyCallBack.handleMetadataEntry(log);
-                        } else {
-                            int batchSize = logs.size();
-                            for (int batchIndex = 0; batchIndex < batchSize; batchIndex++){
-                                PendingAckMetadataEntry log = logs.get(batchIndex);
+                        try {
+                            currentLoadPosition = PositionFactory.create(entry.getLedgerId(), entry.getEntryId());
+                            List<PendingAckMetadataEntry> logs = deserializeEntry(entry);
+                            if (logs.isEmpty()){
+                                continue;
+                            } else if (logs.size() == 1){
+                                currentIndexLag.incrementAndGet();
+                                PendingAckMetadataEntry log = logs.get(0);
+                                handleMetadataEntry(PositionFactory.create(entry.getLedgerId(), entry.getEntryId()),
+                                        log);
                                 pendingAckReplyCallBack.handleMetadataEntry(log);
+                            } else {
+                                int batchSize = logs.size();
+                                for (int batchIndex = 0; batchIndex < batchSize; batchIndex++){
+                                    PendingAckMetadataEntry log = logs.get(batchIndex);
+                                    pendingAckReplyCallBack.handleMetadataEntry(log);
+                                }
+                                currentIndexLag.addAndGet(batchSize);
+                                handleMetadataEntry(PositionFactory.create(entry.getLedgerId(), entry.getEntryId()),
+                                        logs);
                             }
-                            currentIndexLag.addAndGet(batchSize);
-                            handleMetadataEntry(PositionFactory.create(entry.getLedgerId(), entry.getEntryId()), logs);
+                        } finally {
+                            entry.release();
                         }
-                        entry.release();
                         clearUselessLogData();
                     } else {
+                        // Covers a read that was issued but whose callback never arrives: no failure is
+                        // ever delivered, so readEntriesFailed cannot notice the close. A cursor closed
+                        // between two reads takes a different path -- that read fails synchronously in
+                        // ManagedCursorImpl#asyncReadEntriesWithSkip, which clears isReadable and ends
+                        // the loop through the existing read failure handling.
+                        if (cursor.isClosed()) {
+                            stopReplay();
+                            pendingAckReplyCallBack.replayFailed(new ManagedLedgerException
+                                    .CursorAlreadyClosedException("MLPendingAckStore cursor was closed "
+                                    + "while replaying."));
+                            log.warn("MLPendingAckStore cursor was closed while replaying, close replay thread");
+                            return;
+                        }
                         try {
                             Thread.sleep(1);
                         } catch (InterruptedException e) {
-                            if (Thread.interrupted()) {
-                                log.error("[{}]Transaction pending "
-                                        + "replay thread interrupt!", managedLedger.getName(), e);
-                            }
+                            // Restore the interrupt flag and stop. The replay is incomplete, so it must not
+                            // be reported as successful: replayFailed() lets PendingAckHandleImpl decide
+                            // whether to retry instead of leaving the handle Ready with partial state.
+                            Thread.currentThread().interrupt();
+                            stopReplay();
+                            pendingAckReplyCallBack.replayFailed(e);
+                            log.warn("[{}] Transaction pending ack replay thread was interrupted",
+                                    managedLedger.getName(), e);
+                            return;
                         }
                     }
                 }
             } catch (Exception e) {
+                stopReplay();
                 pendingAckReplyCallBack.replayFailed(e);
                 log.error("[{}] Pending ack recover fail!", subManagedCursor.getManagedLedger().getName(), e);
                 return;
             }
+            stopReplay();
             pendingAckReplyCallBack.replayComplete();
         }
+
+        /**
+         * Ends the replay. Entries that were read but never processed are released, and any read that is
+         * still in flight will release its entries itself instead of queueing them, so nothing is left
+         * holding a buffer once the replay thread is gone. Called on every exit from {@link #run()}.
+         */
+        private void stopReplay() {
+            for (Entry entry : fillEntryQueueCallback.stopAndDrain()) {
+                entry.release();
+            }
+        }
     }
 
     private List<PendingAckMetadataEntry> deserializeEntry(Entry entry){
@@ -471,6 +507,13 @@ public class MLPendingAckStore implements PendingAckStore {
         private volatile boolean isReadable = true;
         private final AtomicLong outstandingReadsRequests = new AtomicLong(0);
         private static final int NUMBER_OF_PER_READ_ENTRY = 100;
+        /**
+         * Guards {@link #stopped} against {@link #readEntriesComplete}. A read can still be in flight when
+         * the replay ends, and its completion runs on a managed ledger thread, so handing ownership of the
+         * entries over has to be atomic with respect to enqueuing them.
+         */
+        private final Object stopLock = new Object();
+        private boolean stopped;
 
         boolean fillQueue() {
             if (entryQueue.size() + NUMBER_OF_PER_READ_ENTRY < entryQueue.capacity()
@@ -478,6 +521,25 @@ public class MLPendingAckStore implements PendingAckStore {
                 if (cursor.hasMoreEntries()) {
                     outstandingReadsRequests.incrementAndGet();
                     readAsync(NUMBER_OF_PER_READ_ENTRY, this);
+                } else if (entryQueue.size() == 0) {
+                    // Nothing left to read and everything read so far has been processed: the replay is
+                    // done. The loop condition in PendingAckReplay cannot detect this on its own because
+                    // it compares lastConfirmedEntry -- a snapshot taken when this store was created --
+                    // against currentLoadPosition, which starts at the cursor's mark-delete position,
+                    // while whether anything can still be read is decided by the cursor's read position.
+                    // Those two can disagree permanently, e.g. when the ledger holding the mark-delete
+                    // position was trimmed and the cursor was recovered onto a later ledger. Entries
+                    // below the mark-delete position have already been applied, so completing here is
+                    // correct. TopicTransactionBuffer's equivalent loop was fixed the same way in
+                    // https://github.com/apache/pulsar/pull/13739
+                    if (log.isDebugEnabled()) {
+                        log.debug("[{}] Pending ack replay stopped before reaching the last confirmed entry "
+                                        + "because the cursor has nothing left to read. lastConfirmedEntry={}, "
+                                        + "currentLoadPosition={}, markDeletePosition={}, readPosition={}",
+                                managedLedger.getName(), lastConfirmedEntry, currentLoadPosition,
+                                cursor.getMarkDeletedPosition(), cursor.getReadPosition());
+                    }
+                    isReadable = false;
                 }
             }
             return isReadable;
@@ -485,19 +547,49 @@ public class MLPendingAckStore implements PendingAckStore {
 
         @Override
         public void readEntriesComplete(List<Entry> entries, Object ctx) {
-            entryQueue.fill(new MessagePassingQueue.Supplier<Entry>() {
-                private int i = 0;
-                @Override
-                public Entry get() {
-                    Entry entry = entries.get(i);
-                    i++;
-                    return entry;
+            List<Entry> entriesToRelease = null;
+            synchronized (stopLock) {
+                if (stopped) {
+                    // The replay already finished, so nothing will ever consume these.
+                    entriesToRelease = entries;
+                } else {
+                    int filled = entryQueue.fill(new MessagePassingQueue.Supplier<Entry>() {
+                        private int i = 0;
+                        @Override
+                        public Entry get() {
+                            Entry entry = entries.get(i);
+                            i++;
+                            return entry;
+                        }
+                    }, entries.size());
+                    if (filled < entries.size()) {
+                        entriesToRelease = entries.subList(filled, entries.size());
+                    }
                 }
-            }, entries.size());
-
+            }
+            // Released outside the lock: releasing an entry can run a deallocation callback.
+            if (entriesToRelease != null) {
+                entriesToRelease.forEach(Entry::release);
+            }
             outstandingReadsRequests.decrementAndGet();
         }
 
+        /**
+         * Stops accepting entries and returns everything still queued, so that the replay thread can
+         * release them. Must only be called by the replay thread, which is the queue's single consumer.
+         */
+        List<Entry> stopAndDrain() {
+            List<Entry> drained = new ArrayList<>();
+            synchronized (stopLock) {
+                stopped = true;
+                Entry entry;
+                while ((entry = entryQueue.poll()) != null) {
+                    drained.add(entry);
+                }
+            }
+            return drained;
+        }
+
         @Override
         public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
             if (managedLedger.getConfig().isAutoSkipNonRecoverableData()
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java
index bc4c74ab3b16..bc85802c524a 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleImpl.java
@@ -187,20 +187,29 @@ public class PendingAckHandleImpl extends PendingAckHandleState implements Pendi
     }
 
     private void initPendingAckStore() {
-        if (changeToInitializingState()) {
+        CompletableFuture<PendingAckStore> storeFuture;
+        synchronized (this) {
+            if (!changeToInitializingState()) {
+                return;
+            }
+            // Publish ownership before calling the provider so closeAsync() also waits for a store
+            // whose creation has started but whose future has not yet been returned.
+            storeFuture = new CompletableFuture<>();
+            this.pendingAckStoreFuture = storeFuture;
+        }
+        storeFuture.thenAccept(pendingAckStore -> {
             if (!checkIfClose()) {
-                this.pendingAckStoreFuture =
-                        pendingAckStoreProvider.newPendingAckStore(persistentSubscription);
-                this.pendingAckStoreFuture.thenAccept(pendingAckStore -> {
-                    recoverTime.setRecoverStartTime(System.currentTimeMillis());
-                    pendingAckStore.replayAsync(this, internalPinnedExecutor);
-                }).exceptionallyAsync(e -> {
-                    // Handling the exceptions in `exceptionHandleFuture`,
-                    // it will be helpful to make the exception handling clearer.
-                    exceptionHandleFuture(e.getCause());
-                    return null;
-                }, internalPinnedExecutor);
+                recoverTime.setRecoverStartTime(System.currentTimeMillis());
+                pendingAckStore.replayAsync(this, internalPinnedExecutor);
             }
+        }).exceptionallyAsync(e -> {
+            exceptionHandleFuture(FutureUtil.unwrapCompletionException(e));
+            return null;
+        }, internalPinnedExecutor);
+        try {
+            FutureUtil.completeAfter(storeFuture, pendingAckStoreProvider.newPendingAckStore(persistentSubscription));
+        } catch (Exception e) {
+            storeFuture.completeExceptionally(e);
         }
     }
 
@@ -958,8 +967,10 @@ public class PendingAckHandleImpl extends PendingAckHandleState implements Pendi
     }
 
     public void exceptionHandleFuture(Throwable t) {
-        if (isRetryableException(t)) {
-            this.state = State.None;
+        // Preserve Close atomically: closeAsync() changes the state outside the handle monitor.
+        // Resetting it after a separate close check could allow initPendingAckStore() to reopen the store.
+        boolean retryable = isRetryableException(t);
+        if (retryable && changeToNoneStateIfNotClosed()) {
             long retryTime = backoff.next().toMillis();
             log.warn("[{}][{}] Failed to init transaction pending ack. It will be retried in {} Ms",
                     persistentSubscription.getTopic().getName(), subName, retryTime, t);
@@ -971,8 +982,8 @@ public class PendingAckHandleImpl extends PendingAckHandleState implements Pendi
         changeToErrorState();
         // ToDo: Add a new serverError `TransactionComponentLoadFailedException`
         //  and before that a `Unknown` will be returned first.
-        this.pendingAckStoreFuture = FutureUtil.failedFuture(new BrokerServiceException(
-                        String.format("[%s][%s] Failed to init transaction pending ack.", topicName, subName)));
+        // Keep the original store future, including for non-retryable replay failures, so closeAsync()
+        // can still close the store. Report initialization failure through the handle future below.
         final boolean completedNow = this.pendingAckHandleCompletableFuture.completeExceptionally(
                 new BrokerServiceException(
                 String.format("[%s][%s] Failed to init transaction pending ack.", topicName, subName)));
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleState.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleState.java
index 96653244b764..210cc85f1384 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleState.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/PendingAckHandleState.java
@@ -59,8 +59,13 @@ public abstract class PendingAckHandleState {
         STATE_UPDATER.set(this, State.Close);
     }
 
+    protected boolean changeToNoneStateIfNotClosed() {
+        return STATE_UPDATER.updateAndGet(this, current -> current == State.Close ? current : State.None)
+                != State.Close;
+    }
+
     protected void changeToErrorState() {
-        STATE_UPDATER.set(this, State.Error);
+        STATE_UPDATER.updateAndGet(this, current -> current == State.Close ? current : State.Error);
     }
 
     public boolean checkIfReady() {
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreTest.java
index 14e1ceade503..90f9cccd3c7b 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreTest.java
@@ -18,29 +18,56 @@
  */
 package org.apache.pulsar.broker.transaction.pendingack.impl;
 
+import static org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriter.BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER;
+import static org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriter.BATCHED_ENTRY_DATA_PREFIX_VERSION;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
 import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.timeout;
+import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
+import io.netty.util.Timer;
+import io.netty.util.concurrent.DefaultThreadFactory;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
 import org.apache.bookkeeper.mledger.ManagedCursor;
+import org.apache.bookkeeper.mledger.ManagedLedger;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
 import org.apache.bookkeeper.mledger.Position;
 import org.apache.bookkeeper.mledger.PositionFactory;
+import org.apache.bookkeeper.mledger.impl.EntryImpl;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
 import org.apache.pulsar.broker.ServiceConfiguration;
 import org.apache.pulsar.broker.service.Subscription;
 import org.apache.pulsar.broker.service.persistent.PersistentSubscription;
 import org.apache.pulsar.broker.service.persistent.PersistentTopic;
 import org.apache.pulsar.broker.transaction.TransactionTestBase;
+import org.apache.pulsar.broker.transaction.pendingack.PendingAckReplyCallBack;
+import org.apache.pulsar.broker.transaction.pendingack.PendingAckStore;
+import org.apache.pulsar.broker.transaction.pendingack.TransactionPendingAckStoreProvider;
 import org.apache.pulsar.broker.transaction.util.LogIndexLagBackoff;
 import org.apache.pulsar.client.api.transaction.TxnID;
 import org.apache.pulsar.common.api.proto.CommandAck;
@@ -49,6 +76,7 @@ import org.apache.pulsar.common.naming.SystemTopicNames;
 import org.apache.pulsar.common.naming.TopicName;
 import org.apache.pulsar.common.util.Codec;
 import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.transaction.coordinator.impl.DisabledTxnLogBufferedWriterMetricsStats;
 import org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriterConfig;
 import org.awaitility.Awaitility;
 import org.mockito.invocation.InvocationOnMock;
@@ -170,6 +198,340 @@ public class MLPendingAckStoreTest extends TransactionTestBase {
         closePendingAckStoreWithRetry(store);
     }
 
+    @Test
+    public void testReplayFailurePreservesClosedHandle() throws Exception {
+        PendingAckHandleImpl handle = new PendingAckHandleImpl(persistentSubscriptionMock);
+        handle.pendingAckHandleFuture().get(10, TimeUnit.SECONDS);
+        handle.closeAsync().get(10, TimeUnit.SECONDS);
+
+        // Multiple late failures must not turn Close into Error and then enable a retry from Error.
+        for (int i = 0; i < 2; i++) {
+            handle.exceptionHandleFuture(new ManagedLedgerException.CursorAlreadyClosedException("closed"));
+            Assert.assertEquals(handle.getState(), PendingAckHandleState.State.Close);
+            Assert.assertFalse(handle.changeToNoneStateIfNotClosed());
+            Assert.assertFalse(handle.changeToInitializingState());
+        }
+    }
+
+    @DataProvider(name = "lateReplayFailures")
+    public Object[][] lateReplayFailures() {
+        return new Object[][] {
+                {new ManagedLedgerException.CursorAlreadyClosedException("closed")},
+                {new ManagedLedgerException.ManagedLedgerFencedException()}
+        };
+    }
+
+    @Test(dataProvider = "lateReplayFailures")
+    public void testReplayFailureDoesNotHideStoreFromConcurrentClose(ManagedLedgerException failure) throws Exception {
+        // Create the log so the handle opens a real store during initialization.
+        closePendingAckStoreWithRetry(createPendingAckStore(new TxnLogBufferedWriterConfig()));
+        PendingAckHandleImpl handle = new PendingAckHandleImpl(persistentSubscriptionMock);
+        handle.pendingAckHandleFuture().get(10, TimeUnit.SECONDS);
+        ManagedLedgerImpl ledger = (ManagedLedgerImpl) handle.getStoreManageLedger().get(10, TimeUnit.SECONDS);
+        ExecutorService closeExecutor = Executors.newSingleThreadExecutor(
+                new DefaultThreadFactory("pending-ack-close"));
+        try {
+            Future<CompletableFuture<Void>> closeResult;
+            synchronized (handle) {
+                closeResult = closeExecutor.submit(handle::closeAsync);
+                // closeAsync publishes Close before taking this monitor. Hold it as replayFailed does,
+                // ensuring the failure runs before closeAsync can retrieve the store future.
+                Awaitility.await().atMost(10, TimeUnit.SECONDS).until(handle::checkIfClose);
+                handle.exceptionHandleFuture(failure);
+            }
+            closeResult.get(10, TimeUnit.SECONDS).get(10, TimeUnit.SECONDS);
+            Assert.assertEquals(ledger.getState(), ManagedLedgerImpl.State.Closed,
+                    "The store must still be closed after a late replay failure");
+        } finally {
+            closeExecutor.shutdownNow();
+            ledger.close();
+        }
+    }
+
+    @Test
+    public void testCloseDuringPendingAckStoreCreation() throws Exception {
+        PersistentSubscription subscription = mock(PersistentSubscription.class, RETURNS_DEEP_STUBS);
+        doReturn(persistentSubscriptionMock.getTopicName()).when(subscription).getTopicName();
+        when(subscription.getName()).thenReturn("close-during-store-creation");
+        var pulsar = subscription.getTopic().getBrokerService().getPulsar();
+        when(pulsar.getTransactionExecutorProvider().getExecutor(any(PendingAckHandleImpl.class)))
+                .thenReturn(internalPinnedExecutor);
+        TransactionPendingAckStoreProvider provider = mock(TransactionPendingAckStoreProvider.class);
+        when(pulsar.getTransactionPendingAckStoreProvider()).thenReturn(provider);
+        CompletableFuture<Boolean> initialized = new CompletableFuture<>();
+        when(provider.checkInitializedBefore(subscription)).thenReturn(initialized);
+        PendingAckStore store = mock(PendingAckStore.class);
+        CompletableFuture<Void> storeClosed = new CompletableFuture<>();
+        when(store.closeAsync()).thenReturn(storeClosed);
+        CompletableFuture<PendingAckStore> storeCreated = new CompletableFuture<>();
+        CompletableFuture<CompletableFuture<Void>> closeStarted = new CompletableFuture<>();
+        PendingAckHandleImpl handle = new PendingAckHandleImpl(subscription);
+        when(provider.newPendingAckStore(subscription)).thenAnswer(invocation -> {
+            // Close after init's state check, but before the provider returns its future.
+            closeStarted.complete(handle.closeAsync());
+            return storeCreated;
+        });
+        try {
+            initialized.complete(true);
+            CompletableFuture<Void> closed = closeStarted.get(10, TimeUnit.SECONDS);
+            assertThat(closed).as("Close must wait for the store being created").isNotDone();
+            storeCreated.complete(store);
+            verify(store, timeout(10000)).closeAsync();
+            assertThat(closed).as("Close must wait for store cleanup").isNotDone();
+            storeClosed.complete(null);
+            closed.get(10, TimeUnit.SECONDS);
+            handle.getInternalPinnedExecutor().submit(() -> { }).get(10, TimeUnit.SECONDS);
+            verify(store, never()).replayAsync(any(), any());
+            assertThat(handle.getState()).isEqualTo(PendingAckHandleState.State.Close);
+        } finally {
+            storeCreated.complete(store);
+            storeClosed.complete(null);
+            handle.closeAsync().get(10, TimeUnit.SECONDS);
+        }
+    }
+
+    @DataProvider(name = "replayEntryPayloads")
+    public Object[][] replayEntryPayloads() {
+        return new Object[][] {
+                {new byte[] {0}, true}, // Too short to decode the entry prefix.
+                {ByteBuffer.allocate(4).putShort(BATCHED_ENTRY_DATA_PREFIX_MAGIC_NUMBER)
+                        .putShort(BATCHED_ENTRY_DATA_PREFIX_VERSION).array(), false} // Empty batch.
+        };
+    }
+
+    @Test(dataProvider = "replayEntryPayloads")
+    public void testReplayReleasesPolledAndQueuedEntries(byte[] payload, boolean expectFailure) {
+        ManagedCursor cursor = createReplayCursorMock();
+        when(cursor.hasMoreEntries()).thenReturn(true);
+        EntryImpl first = EntryImpl.create(5, 10, payload);
+        EntryImpl second = EntryImpl.create(5, 11, payload);
+        doAnswer(invocation -> {
+            AsyncCallbacks.ReadEntriesCallback callback = invocation.getArgument(1);
+            callback.readEntriesComplete(List.of(first, second), null);
+            return null;
+        }).when(cursor).asyncReadEntries(anyInt(), any(), any(), any());
+        try {
+            MLPendingAckStore store = createPendingAckStoreForReplay(cursor);
+            PendingAckReplyCallBack callback = mock(PendingAckReplyCallBack.class);
+            store.new PendingAckReplay(callback).run();
+            if (expectFailure) {
+                verify(callback).replayFailed(any(IndexOutOfBoundsException.class));
+                verify(callback, never()).replayComplete();
+            } else {
+                verify(callback).replayComplete();
+                verify(callback, never()).replayFailed(any());
+            }
+            Assert.assertEquals(first.refCnt(), 0, "The polled entry must be released");
+            Assert.assertEquals(second.refCnt(), 0, "The queued entry must be released");
+        } finally {
+            if (first.refCnt() > 0) {
+                first.release();
+            }
+            if (second.refCnt() > 0) {
+                second.release();
+            }
+        }
+    }
+
+    /**
+     * Builds a store whose replay loop starts in the state described by the parameters, without going
+     * through the provider: the replay guard compares {@code lastConfirmedEntry} (a snapshot of the
+     * managed ledger's last confirmed entry, taken here) against a load position seeded from the
+     * cursor's mark-delete position, while whether anything can still be read is decided separately by
+     * {@link ManagedCursor#hasMoreEntries()}.
+     */
+    private MLPendingAckStore createPendingAckStoreForReplay(ManagedCursor cursor) {
+        ManagedLedger managedLedger = mock(ManagedLedger.class);
+        when(managedLedger.getName()).thenReturn("test-pending-ack-log");
+        // Deliberately ahead of the cursor's mark-delete position, so the replay loop's guard stays true.
+        when(managedLedger.getLastConfirmedEntry()).thenReturn(PositionFactory.create(5, 10));
+        TxnLogBufferedWriterConfig config = new TxnLogBufferedWriterConfig();
+        config.setBatchEnabled(false);
+        // The replay failure log reads the subscription cursor's managed ledger name.
+        ManagedCursor subManagedCursor = mock(ManagedCursor.class);
+        when(subManagedCursor.getManagedLedger()).thenReturn(managedLedger);
+        return new MLPendingAckStore(managedLedger, cursor, subManagedCursor, 1, config,
+                mock(Timer.class), DisabledTxnLogBufferedWriterMetricsStats.DISABLED_BUFFERED_WRITER_METRICS,
+                internalPinnedExecutor);
+    }
+
+    private ManagedCursor createReplayCursorMock() {
+        ManagedCursor cursor = mock(ManagedCursor.class);
+        when(cursor.getName()).thenReturn("test-pending-ack-cursor");
+        when(cursor.getMarkDeletedPosition()).thenReturn(PositionFactory.create(1, 0));
+        return cursor;
+    }
+
+    /**
+     * The replay loop must finish once the cursor has nothing left to read, even though its termination
+     * guard still believes there is work to do. Those two are measured from different positions -- the
+     * guard from the cursor's mark-delete position, the read gate from its read position -- so they can
+     * disagree permanently, for instance after the ledger holding the mark-delete position was trimmed
+     * and the cursor was recovered onto a later ledger. Before the fix the loop spun on Thread.sleep(1)
+     * forever, holding its executor thread and starving every other subscription hashed onto it.
+     */
+    @Test
+    public void testReplayCompletesWhenCursorHasNoMoreEntries() throws Exception {
+        ManagedCursor cursor = createReplayCursorMock();
+        when(cursor.isClosed()).thenReturn(false);
+        // No entry between the mark-delete position and the snapshot can be read any more.
+        when(cursor.hasMoreEntries()).thenReturn(false);
+        MLPendingAckStore pendingAckStore = createPendingAckStoreForReplay(cursor);
+
+        ExecutorService replayExecutor = Executors.newSingleThreadExecutor();
+        try {
+            PendingAckHandleImpl pendingAckHandle = mock(PendingAckHandleImpl.class);
+            when(pendingAckHandle.getInternalPinnedExecutor()).thenReturn(replayExecutor);
+            when(pendingAckHandle.changeToReadyState()).thenReturn(true);
+            CountDownLatch replayFinished = new CountDownLatch(1);
+            doAnswer(invocation -> {
+                replayFinished.countDown();
+                return null;
+            }).when(pendingAckHandle).completeHandleFuture();
+
+            pendingAckStore.replayAsync(pendingAckHandle, replayExecutor);
+
+            // The replay thread is shared by every subscription hashed onto it, so a task queued behind a
+            // finished replay must still get to run. This is what the stall actually broke.
+            CountDownLatch queuedBehindReplay = new CountDownLatch(1);
+            replayExecutor.execute(queuedBehindReplay::countDown);
+
+            Assert.assertTrue(replayFinished.await(10, TimeUnit.SECONDS),
+                    "Replay never completed: the replay loop did not terminate");
+            Assert.assertTrue(queuedBehindReplay.await(10, TimeUnit.SECONDS),
+                    "A task queued behind the replay never ran: the replay thread was not released");
+        } finally {
+            replayExecutor.shutdownNow();
+        }
+    }
+
+    /**
+     * A read whose completion never arrives leaves the replay loop waiting with no way to make progress.
+     * Closing the cursor -- which is what unloading the topic does -- must stop it, so that the replay
+     * thread cannot outlive the subscription it belongs to.
+     */
+    @Test
+    public void testReplayStopsWhenCursorIsClosedWhileWaitingForEntries() throws Exception {
+        ManagedCursor cursor = createReplayCursorMock();
+        when(cursor.hasMoreEntries()).thenReturn(true);
+        AtomicBoolean cursorClosed = new AtomicBoolean(false);
+        when(cursor.isClosed()).thenAnswer(invocation -> cursorClosed.get());
+        CountDownLatch readIssued = new CountDownLatch(1);
+        // Drop the read: neither readEntriesComplete nor readEntriesFailed is ever invoked, so the loop
+        // keeps waiting for entries that never arrive.
+        doAnswer(invocation -> {
+            readIssued.countDown();
+            return null;
+        }).when(cursor).asyncReadEntries(anyInt(), any(), any(), any());
+        MLPendingAckStore pendingAckStore = createPendingAckStoreForReplay(cursor);
+
+        ExecutorService replayExecutor = Executors.newSingleThreadExecutor();
+        try {
+            PendingAckHandleImpl pendingAckHandle = mock(PendingAckHandleImpl.class);
+            when(pendingAckHandle.getInternalPinnedExecutor()).thenReturn(replayExecutor);
+
+            pendingAckStore.replayAsync(pendingAckHandle, replayExecutor);
+            Assert.assertTrue(readIssued.await(10, TimeUnit.SECONDS), "Replay never issued a read");
+
+            cursorClosed.set(true);
+
+            verify(pendingAckHandle, timeout(TimeUnit.SECONDS.toMillis(10)))
+                    .exceptionHandleFuture(any(ManagedLedgerException.CursorAlreadyClosedException.class));
+            // The replay must have released the thread it was holding.
+            CountDownLatch queuedBehindReplay = new CountDownLatch(1);
+            replayExecutor.execute(queuedBehindReplay::countDown);
+            Assert.assertTrue(queuedBehindReplay.await(10, TimeUnit.SECONDS),
+                    "A task queued behind the replay never ran: the replay thread was not released");
+        } finally {
+            replayExecutor.shutdownNow();
+        }
+    }
+
+    /**
+     * A read can still be in flight when the replay ends, and its completion runs on a managed ledger
+     * thread. Entries delivered after the replay has gone must be released by the callback itself,
+     * because nothing will ever take them off the queue.
+     */
+    @Test
+    public void testEntriesDeliveredAfterReplayEndedAreReleased() throws Exception {
+        ManagedCursor cursor = createReplayCursorMock();
+        when(cursor.hasMoreEntries()).thenReturn(true);
+        AtomicBoolean cursorClosed = new AtomicBoolean(false);
+        when(cursor.isClosed()).thenAnswer(invocation -> cursorClosed.get());
+        CountDownLatch readIssued = new CountDownLatch(1);
+        AtomicReference<AsyncCallbacks.ReadEntriesCallback> readCallback = new AtomicReference<>();
+        // Capture the callback and never complete it, so the read is still in flight when the replay ends.
+        doAnswer(invocation -> {
+            readCallback.set(invocation.getArgument(1));
+            readIssued.countDown();
+            return null;
+        }).when(cursor).asyncReadEntries(anyInt(), any(), any(), any());
+        MLPendingAckStore pendingAckStore = createPendingAckStoreForReplay(cursor);
+
+        ExecutorService replayExecutor = Executors.newSingleThreadExecutor();
+        try {
+            PendingAckHandleImpl pendingAckHandle = mock(PendingAckHandleImpl.class);
+            when(pendingAckHandle.getInternalPinnedExecutor()).thenReturn(internalPinnedExecutor);
+
+            pendingAckStore.replayAsync(pendingAckHandle, replayExecutor);
+            Assert.assertTrue(readIssued.await(10, TimeUnit.SECONDS), "Replay never issued a read");
+
+            // End the replay while the read is still outstanding.
+            cursorClosed.set(true);
+            verify(pendingAckHandle, timeout(TimeUnit.SECONDS.toMillis(10)))
+                    .exceptionHandleFuture(any(ManagedLedgerException.CursorAlreadyClosedException.class));
+
+            // The read now completes, far too late for the replay to consume anything.
+            Entry first = mock(Entry.class);
+            Entry second = mock(Entry.class);
+            readCallback.get().readEntriesComplete(List.of(first, second), null);
+
+            verify(first).release();
+            verify(second).release();
+        } finally {
+            replayExecutor.shutdownNow();
+        }
+    }
+
+    /**
+     * Shutting down the replay executor must end the replay rather than leaving the thread behind. The
+     * replay is incomplete at that point, so it must be reported as failed and not as complete --
+     * otherwise the handle would be marked Ready with only part of the pending ack state applied.
+     */
+    @Test
+    public void testReplayStopsWhenInterrupted() throws Exception {
+        ManagedCursor cursor = createReplayCursorMock();
+        when(cursor.isClosed()).thenReturn(false);
+        when(cursor.hasMoreEntries()).thenReturn(true);
+        CountDownLatch readIssued = new CountDownLatch(1);
+        // Drop the read, so the replay is waiting for entries that never arrive when it is interrupted.
+        doAnswer(invocation -> {
+            readIssued.countDown();
+            return null;
+        }).when(cursor).asyncReadEntries(anyInt(), any(), any(), any());
+        MLPendingAckStore pendingAckStore = createPendingAckStoreForReplay(cursor);
+
+        ExecutorService replayExecutor = Executors.newSingleThreadExecutor();
+        try {
+            PendingAckHandleImpl pendingAckHandle = mock(PendingAckHandleImpl.class);
+            when(pendingAckHandle.getInternalPinnedExecutor()).thenReturn(internalPinnedExecutor);
+
+            pendingAckStore.replayAsync(pendingAckHandle, replayExecutor);
+            Assert.assertTrue(readIssued.await(10, TimeUnit.SECONDS), "Replay never issued a read");
+
+            replayExecutor.shutdownNow();
+
+            Assert.assertTrue(replayExecutor.awaitTermination(10, TimeUnit.SECONDS),
+                    "The replay thread did not stop after shutdownNow()");
+            verify(pendingAckHandle, timeout(TimeUnit.SECONDS.toMillis(10)))
+                    .exceptionHandleFuture(any(InterruptedException.class));
+            // An interrupted replay is incomplete and must never be reported as successful.
+            verify(pendingAckHandle, never()).completeHandleFuture();
+        } finally {
+            replayExecutor.shutdownNow();
+        }
+    }
+
     /**
      * Overridden cases:
      *   1. Batched write and replay with batched feature.


(pulsar) 07/15: [fix][test] Preserve class loading in broker interceptor fixtures (#26567)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit e9954e1506f1e434b3cd2b1e1e7621d988d271c0
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Sep 13 08:29:44 2026 +0300

    [fix][test] Preserve class loading in broker interceptor fixtures (#26567)
    
    (cherry picked from commit f3c647fe988063a17000abd1303792f6dfa72534)
---
 .../org/apache/pulsar/broker/intercept/BrokerInterceptorTest.java  | 7 ++++++-
 .../pulsar/broker/intercept/ExceptionsBrokerInterceptorTest.java   | 7 ++++++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorTest.java
index d1fd46068c19..1abe2973a51e 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorTest.java
@@ -18,12 +18,15 @@
  */
 package org.apache.pulsar.broker.intercept;
 
+import static org.mockito.AdditionalAnswers.delegatesTo;
 import static org.mockito.ArgumentMatchers.same;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
 import static org.testng.Assert.assertEquals;
 import java.io.IOException;
+import java.net.URL;
+import java.net.URLClassLoader;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Map;
@@ -93,7 +96,9 @@ public class BrokerInterceptorTest extends ProducerConsumerBase {
     @Override
     protected void customizeMainPulsarTestContextBuilder(PulsarTestContext.Builder pulsarTestContextBuilder) {
         HashMap<String, BrokerInterceptorWithClassLoader> brokerInterceptorWithClassLoaderHashMap = new HashMap<>();
-        NarClassLoader narClassLoader = mock(NarClassLoader.class);
+        // The filter chain uses this context class loader for resource and service-provider lookup.
+        NarClassLoader narClassLoader = mock(NarClassLoader.class,
+                delegatesTo(new URLClassLoader(new URL[0], getClass().getClassLoader())));
         BrokerInterceptorWithClassLoader counterBrokerInterceptor =
                 new BrokerInterceptorWithClassLoader(new CounterBrokerInterceptor(), narClassLoader);
         brokerInterceptorWithClassLoaderHashMap.put(CounterBrokerInterceptor.NAME, counterBrokerInterceptor);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/ExceptionsBrokerInterceptorTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/ExceptionsBrokerInterceptorTest.java
index bc18b9ad6794..78747f79d5f6 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/ExceptionsBrokerInterceptorTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/ExceptionsBrokerInterceptorTest.java
@@ -18,10 +18,13 @@
  */
 package org.apache.pulsar.broker.intercept;
 
+import static org.mockito.AdditionalAnswers.delegatesTo;
 import static org.mockito.Mockito.mock;
 import static org.testng.Assert.assertEquals;
 import static org.testng.Assert.assertNotNull;
 import static org.testng.Assert.assertTrue;
+import java.net.URL;
+import java.net.URLClassLoader;
 import java.nio.charset.StandardCharsets;
 import java.util.HashMap;
 import java.util.Map;
@@ -63,7 +66,9 @@ public class ExceptionsBrokerInterceptorTest extends ProducerConsumerBase {
     protected void customizeMainPulsarTestContextBuilder(PulsarTestContext.Builder pulsarTestContextBuilder) {
         Map<String, BrokerInterceptorWithClassLoader> listenerMap = new HashMap<>();
         BrokerInterceptor interceptor = new ExceptionsBrokerInterceptor();
-        NarClassLoader narClassLoader = mock(NarClassLoader.class);
+        // The filter chain uses this context class loader for resource and service-provider lookup.
+        NarClassLoader narClassLoader = mock(NarClassLoader.class,
+                delegatesTo(new URLClassLoader(new URL[0], getClass().getClassLoader())));
         listenerMap.put(interceptorName, new BrokerInterceptorWithClassLoader(interceptor, narClassLoader));
         pulsarTestContextBuilder.brokerInterceptor(new BrokerInterceptors(listenerMap));
     }


(pulsar) 03/15: [fix][admin] Allow defaultNumPartitions for non-partitioned autoTopicCreation override (#25163)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 73d3695880ce342929a3383ffd72830e481c4837
Author: Chimdumebi Nebolisa <13...@users.noreply.github.com>
AuthorDate: Sat Sep 12 14:37:25 2026 -0500

    [fix][admin] Allow defaultNumPartitions for non-partitioned autoTopicCreation override (#25163)
    
    (cherry picked from commit 93aa6b8fe10c45e9070abbc52ff1fe4c0bef8572)
---
 .../apache/pulsar/broker/admin/AdminApi2Test.java  | 48 ++++++++++++++++++++++
 .../SetReplicationClustersValidationTest.java      |  2 +-
 .../data/impl/AutoTopicCreationOverrideImpl.java   |  8 ++--
 .../data/AutoTopicCreationOverrideTest.java        | 24 ++++++++++-
 4 files changed, 77 insertions(+), 5 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
index 5c3a5e35d701..917e26673055 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java
@@ -2554,6 +2554,54 @@ public class AdminApi2Test extends MockedPulsarServiceBaseTest {
             assertTrue(ex.getCause() instanceof NotAcceptableException);
         }
     }
+
+    @Test
+    public void testAutoTopicCreationOverrideNonPartitionedDefaultNumPartitionsIgnored() throws Exception {
+        String tenantName = newUniqueName("prop-xyz2");
+        String namespaceName = tenantName + "/ns-" + System.currentTimeMillis();
+        TenantInfoImpl tenantInfo = new TenantInfoImpl(Set.of("role1", "role2"), Set.of("test"));
+        admin.tenants().createTenant(tenantName, tenantInfo);
+        admin.namespaces().createNamespace(namespaceName, Set.of("test"));
+
+        AutoTopicCreationOverride overridePolicy = AutoTopicCreationOverride.builder()
+                .allowAutoTopicCreation(true)
+                .topicType(TopicType.NON_PARTITIONED.toString())
+                .defaultNumPartitions(1)
+                .build();
+        admin.namespaces().setAutoTopicCreation(namespaceName, overridePolicy);
+
+        AutoTopicCreationOverride storedPolicy = admin.namespaces().getAutoTopicCreation(namespaceName);
+        assertTrue(storedPolicy.isAllowAutoTopicCreation());
+        assertEquals(storedPolicy.getTopicType(), TopicType.NON_PARTITIONED.toString());
+        assertEquals(storedPolicy.getDefaultNumPartitions(), Integer.valueOf(1));
+
+        String topicName = "persistent://" + namespaceName + "/auto-topic-" + UUID.randomUUID();
+        pulsarClient.newProducer().topic(topicName).create().close();
+
+        assertEquals(admin.topics().getPartitionedTopicMetadata(topicName).partitions, 0);
+    }
+
+    @Test
+    public void testAutoTopicCreationOverrideNonPartitionedDefaultNumPartitionsRejected() throws Exception {
+        String tenantName = newUniqueName("prop-xyz2");
+        String namespaceName = tenantName + "/ns-" + System.currentTimeMillis();
+        TenantInfoImpl tenantInfo = new TenantInfoImpl(Set.of("role1", "role2"), Set.of("test"));
+        admin.tenants().createTenant(tenantName, tenantInfo);
+        admin.namespaces().createNamespace(namespaceName, Set.of("test"));
+
+        AutoTopicCreationOverride overridePolicy = AutoTopicCreationOverride.builder()
+                .allowAutoTopicCreation(true)
+                .topicType(TopicType.NON_PARTITIONED.toString())
+                .defaultNumPartitions(5)
+                .build();
+        try {
+            admin.namespaces().setAutoTopicCreation(namespaceName, overridePolicy);
+            fail("Should have failed");
+        } catch (PulsarAdminException e) {
+            assertEquals(e.getStatusCode(), Status.PRECONDITION_FAILED.getStatusCode());
+        }
+    }
+
     @Test
     public void testMaxTopicsPerNamespace() throws Exception {
         restartClusterAfterTest();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SetReplicationClustersValidationTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SetReplicationClustersValidationTest.java
index b9fff4b99d4c..742e00a74f5e 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SetReplicationClustersValidationTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SetReplicationClustersValidationTest.java
@@ -245,7 +245,7 @@ public class SetReplicationClustersValidationTest extends OneWayReplicatorTestBa
             admin1.namespaces().setAutoTopicCreation(namespace, policy1);
             fail("Expected behaviour: Pulsar does not allow setting non-partitioned and a certain partition counts");
         } catch (Exception ex) {
-            assertTrue(ex.getMessage().contains("is not allowed to be set when the type is non-partition"));
+            assertTrue(ex.getMessage().contains("must be null, 0 or 1 when the type is non-partitioned"));
         }
     }
 
diff --git a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/impl/AutoTopicCreationOverrideImpl.java b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/impl/AutoTopicCreationOverrideImpl.java
index 52cf1f1829b8..79410b01fd5a 100644
--- a/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/impl/AutoTopicCreationOverrideImpl.java
+++ b/pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/impl/AutoTopicCreationOverrideImpl.java
@@ -44,6 +44,7 @@ public final class AutoTopicCreationOverrideImpl implements AutoTopicCreationOve
             if (!TopicType.isValidTopicType(override.getTopicType())) {
                 return ValidateResult.fail(String.format("Unknown topic type [%s]", override.getTopicType()));
             }
+
             if (TopicType.PARTITIONED.toString().equals(override.getTopicType())) {
                 if (override.getDefaultNumPartitions() == null) {
                     return ValidateResult.fail("[defaultNumPartitions] cannot be null when the type is partitioned.");
@@ -52,9 +53,10 @@ public final class AutoTopicCreationOverrideImpl implements AutoTopicCreationOve
                     return ValidateResult.fail("[defaultNumPartitions] cannot be less than 1 for partition type.");
                 }
             } else if (TopicType.NON_PARTITIONED.toString().equals(override.getTopicType())) {
-                if (override.getDefaultNumPartitions() != null) {
-                    return ValidateResult.fail("[defaultNumPartitions] is not allowed to be"
-                            + " set when the type is non-partition.");
+                Integer p = override.getDefaultNumPartitions();
+                if (p != null && p != 0 && p != 1) {
+                    return ValidateResult.fail(
+                            "[defaultNumPartitions] must be null, 0 or 1 when the type is non-partitioned.");
                 }
             }
         }
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/AutoTopicCreationOverrideTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/AutoTopicCreationOverrideTest.java
index 89d41547ba96..e7a34cf4505a 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/AutoTopicCreationOverrideTest.java
+++ b/pulsar-common/src/test/java/org/apache/pulsar/common/policies/data/AutoTopicCreationOverrideTest.java
@@ -73,7 +73,28 @@ public class AutoTopicCreationOverrideTest {
     }
 
     @Test
-    public void testNumPartitionsOnNonPartitioned() {
+    public void testNumPartitionsOnNonPartitionedZeroAllowed() {
+        AutoTopicCreationOverride override = AutoTopicCreationOverride.builder()
+                .allowAutoTopicCreation(true)
+                .topicType(TopicType.NON_PARTITIONED.toString())
+                .defaultNumPartitions(0)
+                .build();
+        assertTrue(AutoTopicCreationOverrideImpl.validateOverride(override).isSuccess());
+    }
+
+    @Test
+    public void testNumPartitionsOnNonPartitionedOneAllowed() {
+        AutoTopicCreationOverride override = AutoTopicCreationOverride.builder()
+                .allowAutoTopicCreation(true)
+                .topicType(TopicType.NON_PARTITIONED.toString())
+                .defaultNumPartitions(1)
+                .build();
+        assertTrue(AutoTopicCreationOverrideImpl.validateOverride(override).isSuccess());
+    }
+
+
+    @Test
+    public void testNumPartitionsOnNonPartitionedTooHighRejected() {
         AutoTopicCreationOverride override = AutoTopicCreationOverride.builder()
                 .allowAutoTopicCreation(true)
                 .topicType(TopicType.NON_PARTITIONED.toString())
@@ -81,4 +102,5 @@ public class AutoTopicCreationOverrideTest {
                 .build();
         assertFalse(AutoTopicCreationOverrideImpl.validateOverride(override).isSuccess());
     }
+
 }


(pulsar) 12/15: [fix][test] Cover scheduler shutdown membership-lock deadlock (#26557)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 2a05e1dcdba0f84e7d45a25aa9147f9032b940d5
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Sep 14 01:00:46 2026 +0300

    [fix][test] Cover scheduler shutdown membership-lock deadlock (#26557)
    
    (cherry picked from commit 6075c2b4ddf7fd89a2e08cec4b4b8fe6286ee0af)
---
 pulsar-functions/worker/pom.xml                    |  6 +++
 .../functions/worker/SchedulerManagerTest.java     | 43 ++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/pulsar-functions/worker/pom.xml b/pulsar-functions/worker/pom.xml
index 9daabb673f87..84da1acdb232 100644
--- a/pulsar-functions/worker/pom.xml
+++ b/pulsar-functions/worker/pom.xml
@@ -209,6 +209,12 @@
       <scope>test</scope>
     </dependency>
 
+    <dependency>
+      <groupId>org.awaitility</groupId>
+      <artifactId>awaitility</artifactId>
+      <scope>test</scope>
+    </dependency>
+
   </dependencies>
 
   <build>
diff --git a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
index 87d4e82450d3..076cae8d25f9 100644
--- a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
+++ b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
@@ -26,11 +26,13 @@ import static org.mockito.Mockito.anyString;
 import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertThrows;
 import static org.testng.Assert.assertTrue;
 import com.google.protobuf.InvalidProtocolBufferException;
 import io.netty.util.concurrent.DefaultThreadFactory;
@@ -44,12 +46,15 @@ import java.util.Set;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.ReentrantLock;
 import lombok.extern.slf4j.Slf4j;
 import lombok.val;
 import org.apache.pulsar.client.admin.LongRunningProcessStatus;
@@ -67,6 +72,7 @@ import org.apache.pulsar.functions.runtime.thread.ThreadRuntimeFactory;
 import org.apache.pulsar.functions.runtime.thread.ThreadRuntimeFactoryConfig;
 import org.apache.pulsar.functions.utils.FunctionCommon;
 import org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler;
+import org.awaitility.Awaitility;
 import org.mockito.Mockito;
 import org.mockito.invocation.Invocation;
 import org.testng.Assert;
@@ -155,6 +161,43 @@ public class SchedulerManagerTest {
         this.executor.shutdownNow();
     }
 
+    @Test
+    @SuppressWarnings("unchecked")
+    public void testCloseDoesNotBlockMembershipChecksWhileWaitingForScheduler() throws Exception {
+        doReturn(List.of(WorkerInfo.of("worker-1", "localhost", 5000)))
+                .when(membershipManager).getCurrentMembership();
+        schedulerManager.initialize(producer);
+        ReentrantLock schedulerLock = (ReentrantLock) schedulerManager.getSchedulerLock();
+        AtomicReference<Thread> closingThread = new AtomicReference<>();
+        ExecutorService closeExecutor = Executors.newSingleThreadExecutor(new DefaultThreadFactory("scheduler-close"));
+        Future<?> closeFuture = null;
+        schedulerLock.lock();
+        try {
+            closeFuture = closeExecutor.submit(() -> {
+                closingThread.set(Thread.currentThread());
+                schedulerManager.close();
+            });
+            Awaitility.await().until(() -> closingThread.get() != null
+                    && schedulerLock.hasQueuedThread(closingThread.get()));
+
+            // A scheduler holding schedulerLock must still be able to read membership while close waits.
+            // Use a separate task with a bounded wait so a regression can release the lock during cleanup.
+            executor.submit(() -> assertThrows(SchedulerManager.TooFewWorkersException.class,
+                    () -> schedulerManager.rebalanceIfNotInprogress())).get(5, TimeUnit.SECONDS);
+            verify(producer, never()).close();
+        } finally {
+            schedulerLock.unlock();
+            try {
+                if (closeFuture != null) {
+                    closeFuture.get(5, TimeUnit.SECONDS);
+                }
+            } finally {
+                closeExecutor.shutdownNow();
+            }
+        }
+        verify(producer).close();
+    }
+
     @Test
     public void testSchedule() throws Exception {
 


(pulsar) 11/15: [fix][client] Complete table view refresh after applying messages (#26566)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 078f7b2ab9dcdef6217593347dc42a0956c8cdd6
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Sep 14 00:36:30 2026 +0300

    [fix][client] Complete table view refresh after applying messages (#26566)
    
    (cherry picked from commit 172b7c32c71caf19cf85ad6baf10d7cc7b76be25)
---
 .../apache/pulsar/client/impl/TableViewImpl.java   |  8 +-
 .../pulsar/client/impl/TableViewImplTest.java      | 90 ++++++++++++++++++++++
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TableViewImpl.java b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TableViewImpl.java
index 17b49828eece..998dbafd73c3 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TableViewImpl.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TableViewImpl.java
@@ -204,7 +204,6 @@ public class TableViewImpl<T> implements TableView<T> {
     }
 
     private void handleMessage(Message<T> msg) {
-        lastReadPositions.put(msg.getTopicName(), msg.getMessageId());
         try {
             if (msg.hasKey()) {
                 String key = msg.getKey();
@@ -226,6 +225,8 @@ public class TableViewImpl<T> implements TableView<T> {
                                 key,
                                 cur,
                                 prev);
+                        // The retained value is current before notifying the skipped-message callback.
+                        lastReadPositions.put(msg.getTopicName(), msg.getMessageId());
                         compactionStrategy.handleSkippedMessage(key, cur);
                     }
                 }
@@ -239,6 +240,8 @@ public class TableViewImpl<T> implements TableView<T> {
                             data.put(key, cur);
                         }
 
+                        // Refresh must see the updated table, including when called from a listener.
+                        lastReadPositions.put(msg.getTopicName(), msg.getMessageId());
                         for (BiConsumer<String, T> listener : listeners) {
                             try {
                                 listener.accept(key, cur);
@@ -250,6 +253,9 @@ public class TableViewImpl<T> implements TableView<T> {
                         listenersMutex.unlock();
                     }
                 }
+            } else {
+                // Keyless messages also advance the refresh position.
+                lastReadPositions.put(msg.getTopicName(), msg.getMessageId());
             }
             checkAllFreshTask(msg);
         } finally {
diff --git a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/TableViewImplTest.java b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/TableViewImplTest.java
index c1ab9ae6b62e..9761cc9aed31 100644
--- a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/TableViewImplTest.java
+++ b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/TableViewImplTest.java
@@ -18,14 +18,33 @@
  */
 package org.apache.pulsar.client.impl;
 
+import static org.mockito.Mockito.RETURNS_SELF;
 import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
 import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
 import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
 import org.apache.pulsar.client.api.CryptoKeyReader;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Reader;
+import org.apache.pulsar.client.api.ReaderBuilder;
 import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.client.api.TableView;
+import org.apache.pulsar.common.topics.TopicCompactionStrategy;
 import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
 import org.testng.annotations.Test;
 
 public class TableViewImplTest {
@@ -52,4 +71,75 @@ public class TableViewImplTest {
 
         assertNotNull(tableView);
     }
+    @DataProvider
+    public Object[][] skippedMessage() {
+        return new Object[][]{{false}, {true}};
+    }
+
+    @Test(timeOut = 10_000, dataProvider = "skippedMessage")
+    @SuppressWarnings("unchecked")
+    public void testRefreshWaitsForMessageToBeApplied(boolean skipped) throws Exception {
+        String topic = "persistent://public/default/refresh-applied";
+        PulsarClientImpl client = mock(PulsarClientImpl.class);
+        ReaderBuilder<String> builder = mock(ReaderBuilder.class, RETURNS_SELF);
+        Reader<String> reader = mock(Reader.class);
+        when(client.newReader(Schema.STRING)).thenReturn(builder);
+        when(builder.createAsync()).thenReturn(CompletableFuture.completedFuture(reader));
+        when(reader.closeAsync()).thenReturn(CompletableFuture.completedFuture(null));
+        TopicMessageIdImpl messageId = new TopicMessageIdImpl(topic, new MessageIdImpl(1, 0, -1));
+        when(reader.getLastMessageIdsAsync()).thenReturn(
+                CompletableFuture.completedFuture(List.of()),
+                CompletableFuture.completedFuture(List.of(messageId)));
+        CompletableFuture<Message<String>> nextMessage = new CompletableFuture<>();
+        when(reader.readNextAsync()).thenReturn(nextMessage, new CompletableFuture<>());
+        TableViewConfigurationData conf = new TableViewConfigurationData();
+        conf.setTopicName(topic);
+        TopicCompactionStrategy<String> strategy = mock(TopicCompactionStrategy.class);
+        when(strategy.shouldKeepLeft(any(), any())).thenReturn(skipped);
+        TableViewImpl<String> tableView;
+        try (var strategies = mockStatic(TopicCompactionStrategy.class)) {
+            strategies.when(() -> TopicCompactionStrategy.load(TopicCompactionStrategy.TABLE_VIEW_TAG, null))
+                    .thenReturn(strategy);
+            tableView = new TableViewImpl<>(client, Schema.STRING, conf);
+        }
+        tableView.start().get(5, TimeUnit.SECONDS);
+        AtomicBoolean callbackRefreshCompleted = new AtomicBoolean();
+        tableView.listen((key, value) -> callbackRefreshCompleted.set(tableView.refreshAsync().isDone()));
+        doAnswer(invocation -> {
+            callbackRefreshCompleted.set(tableView.refreshAsync().isDone());
+            return null;
+        }).when(strategy).handleSkippedMessage(any(), any());
+
+        CountDownLatch decoding = new CountDownLatch(1);
+        CountDownLatch applyMessage = new CountDownLatch(1);
+        Message<String> message = mock(Message.class);
+        when(message.getTopicName()).thenReturn(topic);
+        when(message.getMessageId()).thenReturn(messageId);
+        when(message.hasKey()).thenReturn(true);
+        when(message.getKey()).thenReturn("key");
+        when(message.size()).thenReturn(1);
+        when(message.getValue()).thenAnswer(invocation -> {
+            decoding.countDown();
+            assertTrue(applyMessage.await(5, TimeUnit.SECONDS));
+            return "value";
+        });
+        ExecutorService executor = Executors.newSingleThreadExecutor();
+        try {
+            var delivery = executor.submit(() -> nextMessage.complete(message));
+            assertTrue(decoding.await(5, TimeUnit.SECONDS));
+            assertNull(tableView.get("key"));
+            var refresh = tableView.refreshAsync();
+            assertFalse(refresh.isDone(), "Refresh must not finish before the message updates the table");
+            applyMessage.countDown();
+            delivery.get(5, TimeUnit.SECONDS);
+            refresh.get(5, TimeUnit.SECONDS);
+            assertEquals(tableView.get("key"), skipped ? null : "value");
+            assertTrue(callbackRefreshCompleted.get(), "A callback must observe the current message as applied");
+        } finally {
+            applyMessage.countDown();
+            executor.shutdownNow();
+            tableView.close();
+        }
+    }
+
 }


(pulsar) 10/15: [fix][test] Isolate mutable broker fixtures between test methods (#26563)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit d03b88faf8e6122ffc3c848bcef93d4c7a262bb4
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Sep 13 19:29:20 2026 +0300

    [fix][test] Isolate mutable broker fixtures between test methods (#26563)
    
    (cherry picked from commit 0e76cf34d0dd5229a9662618d724e75583c52849)
---
 .../service/BrokerDispatchRateLimiterTest.java     |  8 +--
 .../BrokerServiceAutoTopicCreationTest.java        | 12 ++---
 .../pulsar/broker/service/BrokerServiceTest.java   | 57 +++++++++++-----------
 3 files changed, 35 insertions(+), 42 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerDispatchRateLimiterTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerDispatchRateLimiterTest.java
index 0db1a0f13acf..62f186de94af 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerDispatchRateLimiterTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerDispatchRateLimiterTest.java
@@ -36,19 +36,19 @@ import org.apache.pulsar.client.api.Schema;
 import org.apache.pulsar.client.api.SubscriptionType;
 import org.awaitility.Awaitility;
 import org.testng.Assert;
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
 import org.testng.annotations.Test;
 
 @Test(groups = "broker")
 public class BrokerDispatchRateLimiterTest extends BrokerTestBase {
-    @BeforeClass
+    @BeforeMethod
     @Override
     protected void setup() throws Exception {
         super.baseSetup();
     }
 
-    @AfterClass(alwaysRun = true)
+    @AfterMethod(alwaysRun = true)
     @Override
     protected void cleanup() throws Exception {
         super.internalCleanup();
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
index f8b672489fd5..8cee8cbea8ce 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
@@ -49,31 +49,25 @@ import org.apache.pulsar.common.policies.data.TenantInfoImpl;
 import org.apache.pulsar.common.policies.data.TopicType;
 import org.awaitility.Awaitility;
 import org.testng.Assert;
-import org.testng.annotations.AfterClass;
 import org.testng.annotations.AfterMethod;
-import org.testng.annotations.BeforeClass;
+import org.testng.annotations.BeforeMethod;
 import org.testng.annotations.Test;
 
 @Test(groups = "broker")
 public class BrokerServiceAutoTopicCreationTest extends BrokerTestBase{
 
-    @BeforeClass
+    @BeforeMethod
     @Override
     protected void setup() throws Exception {
         super.baseSetup();
     }
 
-    @AfterClass(alwaysRun = true)
+    @AfterMethod(alwaysRun = true)
     @Override
     protected void cleanup() throws Exception {
         super.internalCleanup();
     }
 
-    @AfterMethod(alwaysRun = true)
-    protected void cleanupTest() throws Exception {
-        pulsar.getAdminClient().namespaces().removeAutoTopicCreation("prop/ns-abc");
-    }
-
     @Test
     public void testAutoNonPartitionedTopicCreation() throws Exception{
         pulsar.getConfiguration().setAllowAutoTopicCreation(true);
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
index 71df23d47232..4c6f37a82bdc 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceTest.java
@@ -2166,37 +2166,36 @@ public class BrokerServiceTest extends BrokerTestBase {
         Set<String> providers = new HashSet<>();
         providers.add("org.apache.pulsar.broker.authentication.AuthenticationProviderTls");
 
-        conf.setAuthenticationEnabled(true);
-        conf.setAuthenticationProviders(providers);
-        conf.setBrokerServicePortTls(Optional.of(0));
-        conf.setWebServicePortTls(Optional.of(0));
-        conf.setTlsCertificateFilePath(BROKER_CERT_FILE_PATH);
-        conf.setTlsKeyFilePath(BROKER_KEY_FILE_PATH);
-        conf.setTlsAllowInsecureConnection(false);
-        conf.setTlsTrustCertsFilePath(CA_CERT_FILE_PATH);
-        conf.setNumExecutorThreadPoolSize(5);
-        restartBroker();
-
-        String authParam = String.format("tlsCertFile:%s,tlsKeyFile:%s", getTlsFileForClient("admin.cert"),
-                getTlsFileForClient("admin.key-pk8"));
-        String authClassName = "org.apache.pulsar.client.impl.auth.AuthenticationTls";
-        ClientConfigurationData conf = new ClientConfigurationData();
-        conf.setServiceUrl(brokerUrlTls.toString());
-        conf.setAuthParams(authParam);
-        conf.setAuthPluginClassName(authClassName);
-        conf.setTlsAllowInsecureConnection(true);
-
-        PulsarClient pulsarClient = null;
         try {
-            pulsarClient = (new ClientBuilderImpl(conf)).build();
-
-            @Cleanup
-            Consumer<byte[]> consumer = pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
-                    .subscribe();
-        } catch (Exception e) {
-            fail("should not fail");
+            conf.setAuthenticationEnabled(true);
+            conf.setAuthenticationProviders(providers);
+            conf.setBrokerServicePortTls(Optional.of(0));
+            conf.setWebServicePortTls(Optional.of(0));
+            conf.setTlsCertificateFilePath(BROKER_CERT_FILE_PATH);
+            conf.setTlsKeyFilePath(BROKER_KEY_FILE_PATH);
+            conf.setTlsAllowInsecureConnection(false);
+            conf.setTlsTrustCertsFilePath(CA_CERT_FILE_PATH);
+            conf.setNumExecutorThreadPoolSize(5);
+            restartBroker();
+
+            String authParam = String.format("tlsCertFile:%s,tlsKeyFile:%s", getTlsFileForClient("admin.cert"),
+                    getTlsFileForClient("admin.key-pk8"));
+            String authClassName = "org.apache.pulsar.client.impl.auth.AuthenticationTls";
+            ClientConfigurationData conf = new ClientConfigurationData();
+            conf.setServiceUrl(brokerUrlTls.toString());
+            conf.setAuthParams(authParam);
+            conf.setAuthPluginClassName(authClassName);
+            conf.setTlsAllowInsecureConnection(true);
+
+            try (PulsarClient pulsarClient = new ClientBuilderImpl(conf).build()) {
+                @Cleanup
+                Consumer<byte[]> consumer = pulsarClient.newConsumer().topic(topicName).subscriptionName(subName)
+                        .subscribe();
+            } catch (Exception e) {
+                fail("should not fail", e);
+            }
         } finally {
-            pulsarClient.close();
+            resetState();
         }
     }
 


(pulsar) 04/15: [fix][common] Handle synchronous failures from future suppliers (#25939)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 3d09c6b3f9d6c02de4dfbd7465f3bec2af5a9359
Author: Radiance' 波波 <39...@users.noreply.github.com>
AuthorDate: Sun Sep 13 04:42:21 2026 +0800

    [fix][common] Handle synchronous failures from future suppliers (#25939)
    
    Co-authored-by: hechengbo <ra...@gmail.com>
    (cherry picked from commit 8efeb05a3ac8625bb7cd43d66b9470587d92a19c)
---
 .../apache/bookkeeper/mledger/util/Futures.java    |  2 +-
 .../bookkeeper/mledger/util/FuturesTest.java       | 45 +++++++++++
 .../org/apache/pulsar/common/util/FutureUtil.java  | 56 ++++++++++----
 .../apache/pulsar/common/util/FutureUtilTest.java  | 89 +++++++++++++++++++++-
 4 files changed, 176 insertions(+), 16 deletions(-)

diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/Futures.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/Futures.java
index f5ad77a71d8c..ecd52d9ce3b1 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/Futures.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/Futures.java
@@ -75,7 +75,7 @@ public class Futures {
                                                             Class<? extends Exception> needRetryExceptionClass,
                                                             int maxRetryTimes) {
         CompletableFuture<T> resultFuture = new CompletableFuture<>();
-        op.get().whenComplete((res, ex) -> {
+        FutureUtil.supplySafely(op).whenComplete((res, ex) -> {
             if (ex == null) {
                 resultFuture.complete(res);
             } else {
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/FuturesTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/FuturesTest.java
new file mode 100644
index 000000000000..d35082912ad4
--- /dev/null
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/util/FuturesTest.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.mledger.util;
+
+import static org.testng.Assert.assertEquals;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.testng.annotations.Test;
+
+public class FuturesTest {
+
+    @Test
+    public void testExecuteWithRetryHandlesSynchronousFailure() throws Exception {
+        AtomicInteger attempts = new AtomicInteger();
+
+        CompletableFuture<String> result = Futures.executeWithRetry(() -> {
+            if (attempts.incrementAndGet() == 1) {
+                throw new CompletionException(new ManagedLedgerException.MetaStoreException("sync fail"));
+            }
+            return CompletableFuture.completedFuture("ok");
+        }, ManagedLedgerException.MetaStoreException.class, 1);
+
+        assertEquals(result.get(2, TimeUnit.SECONDS), "ok");
+        assertEquals(attempts.get(), 2);
+    }
+}
diff --git a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
index 1b43d17d9d12..80a3b30eb9bb 100644
--- a/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
+++ b/pulsar-common/src/main/java/org/apache/pulsar/common/util/FutureUtil.java
@@ -200,6 +200,31 @@ public class FutureUtil {
         return future;
     }
 
+    /**
+     * Invokes a supplier that is expected to return a {@link CompletableFuture}, converting synchronous failures into
+     * a failed future.
+     *
+     * @param supplier the supplier to invoke
+     * @param <T> the result type of the returned future
+     * @return the future returned by the supplier, or a failed future if the supplier is {@code null}, throws,
+     *         or returns {@code null}
+     */
+    public static <T> CompletableFuture<T> supplySafely(Supplier<CompletableFuture<T>> supplier) {
+        if (supplier == null) {
+            return failedFuture(new NullPointerException("Expected Supplier should not be null"));
+        }
+        CompletableFuture<T> future;
+        try {
+            future = supplier.get();
+        } catch (Throwable t) {
+            return failedFuture(t);
+        }
+        if (future == null) {
+            return failedFuture(new NullPointerException("The given supplier returned null, supplier=" + supplier));
+        }
+        return future;
+    }
+
     public static Throwable unwrapCompletionException(Throwable ex) {
         if (ex instanceof CompletionException) {
             return unwrapCompletionException(ex.getCause());
@@ -227,8 +252,9 @@ public class FutureUtil {
         }
 
         /**
-         * @return a {@link CompletableFuture} representing the newly scheduled task,
-         * or one completed exceptionally with {@link NullPointerException} if param is null.
+         * @return a {@link CompletableFuture} representing the newly scheduled task, or the current failed chain when
+         * exceptions are allowed to break the chain. Returns a failed future if the supplier is {@code null}, throws,
+         * or returns {@code null}.
          */
         public synchronized CompletableFuture<T> sequential(Supplier<CompletableFuture<T>> newTask) {
             if (newTask == null) {
@@ -238,11 +264,11 @@ public class FutureUtil {
                 if (sequencerFuture.isCompletedExceptionally() && allowExceptionBreakChain) {
                     return sequencerFuture;
                 }
-                return sequencerFuture = newTask.get();
+                return sequencerFuture = supplySafely(newTask);
             }
             return sequencerFuture = allowExceptionBreakChain
-                    ? sequencerFuture.thenCompose(__ -> newTask.get())
-                    : sequencerFuture.exceptionally(ex -> null).thenCompose(__ -> newTask.get());
+                    ? sequencerFuture.thenCompose(__ -> supplySafely(newTask))
+                    : sequencerFuture.exceptionally(ex -> null).thenCompose(__ -> supplySafely(newTask));
         }
     }
 
@@ -285,8 +311,8 @@ public class FutureUtil {
 
     /**
      * @return a {@link CompletableFuture} representing the asynchronous composition.
-     * The returned future is completed exceptionally with {@link NullPointerException} if one of params is null,
-     * or with {@link RejectedExecutionException} if the task cannot be accepted for execution.
+     * The returned future is completed exceptionally if one of the params is {@code null}, if the supplier throws or
+     * returns {@code null}, or if the executor rejects the task.
      */
     public static <T> @NonNull CompletableFuture<T> composeAsync(Supplier<CompletableFuture<T>> futureSupplier,
                                                                  Executor executor) {
@@ -298,13 +324,15 @@ public class FutureUtil {
         }
         final CompletableFuture<T> future = new CompletableFuture<>();
         try {
-            executor.execute(() -> futureSupplier.get().whenComplete((result, error) -> {
-                if (error != null) {
-                    future.completeExceptionally(error);
-                    return;
-                }
-                future.complete(result);
-            }));
+            executor.execute(() -> {
+                supplySafely(futureSupplier).whenComplete((result, error) -> {
+                    if (error != null) {
+                        future.completeExceptionally(error);
+                        return;
+                    }
+                    future.complete(result);
+                });
+            });
         } catch (RejectedExecutionException ex) {
             future.completeExceptionally(ex);
         }
diff --git a/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java b/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
index ee1483c3a9a8..de24e14608d0 100644
--- a/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
+++ b/pulsar-common/src/test/java/org/apache/pulsar/common/util/FutureUtilTest.java
@@ -21,6 +21,7 @@ package org.apache.pulsar.common.util;
 import static org.testng.Assert.assertEquals;
 import static org.testng.Assert.assertFalse;
 import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertSame;
 import static org.testng.Assert.assertTrue;
 import static org.testng.Assert.fail;
 import java.io.PrintWriter;
@@ -34,6 +35,7 @@ import java.util.concurrent.CompletionException;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import lombok.Cleanup;
 import org.assertj.core.util.Lists;
@@ -183,6 +185,7 @@ public class FutureUtilTest {
         }
     }
 
+    @Test
     public void testSequencer() {
         int concurrentNum = 1000;
         @Cleanup("shutdownNow")
@@ -252,4 +255,88 @@ public class FutureUtilTest {
             Assert.assertEquals(list3.get(i), (Integer) i);
         }
     }
-}
\ No newline at end of file
+
+    @Test
+    public void testSequencerReturnsFailedFutureWhenTaskThrowsSynchronously() throws Exception {
+        FutureUtil.Sequencer<String> sequencer = FutureUtil.Sequencer.create();
+
+        CompletableFuture<String> future = sequencer.sequential(() -> {
+            throw new IllegalStateException("sync fail");
+        });
+
+        try {
+            future.get(2, TimeUnit.SECONDS);
+            fail("Should have failed.");
+        } catch (ExecutionException e) {
+            assertTrue(e.getCause() instanceof IllegalStateException);
+            assertEquals(e.getCause().getMessage(), "sync fail");
+        }
+    }
+
+    @Test
+    public void testSupplySafelyReturnsSupplierFuture() throws Exception {
+        CompletableFuture<String> expected = CompletableFuture.completedFuture("ok");
+        CompletableFuture<String> future = FutureUtil.supplySafely(() -> expected);
+
+        assertSame(future, expected);
+        assertEquals(future.get(2, TimeUnit.SECONDS), "ok");
+    }
+
+    @Test
+    public void testSupplySafelyReturnsFailedFutureWhenSupplierIsNull() throws Exception {
+        CompletableFuture<String> future = FutureUtil.supplySafely(null);
+
+        try {
+            future.get(2, TimeUnit.SECONDS);
+            fail("Should have failed.");
+        } catch (ExecutionException e) {
+            assertTrue(e.getCause() instanceof NullPointerException);
+        }
+    }
+
+    @Test
+    public void testSupplySafelyReturnsFailedFutureWhenSupplierThrowsSynchronously() throws Exception {
+        RuntimeException expected = new IllegalStateException("sync fail");
+
+        CompletableFuture<String> future = FutureUtil.supplySafely(() -> {
+            throw expected;
+        });
+
+        try {
+            future.get(2, TimeUnit.SECONDS);
+            fail("Should have failed.");
+        } catch (ExecutionException e) {
+            assertEquals(e.getCause(), expected);
+        }
+    }
+
+    @Test
+    public void testSupplySafelyReturnsFailedFutureWhenSupplierReturnsNull() throws Exception {
+        CompletableFuture<String> future = FutureUtil.supplySafely(() -> null);
+
+        try {
+            future.get(2, TimeUnit.SECONDS);
+            fail("Should have failed.");
+        } catch (ExecutionException e) {
+            assertTrue(e.getCause() instanceof NullPointerException);
+        }
+    }
+
+    @Test
+    public void testComposeAsyncReturnsFailedFutureWhenSupplierThrowsSynchronously() throws Exception {
+        @Cleanup("shutdownNow")
+        ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
+
+        CompletableFuture<String> future = FutureUtil.composeAsync(() -> {
+            throw new IllegalStateException("sync fail");
+        }, executor);
+
+        try {
+            future.get(2, TimeUnit.SECONDS);
+            fail("Should have failed.");
+        } catch (ExecutionException e) {
+            assertTrue(e.getCause() instanceof IllegalStateException);
+            assertEquals(e.getCause().getMessage(), "sync fail");
+        }
+    }
+}


(pulsar) 09/15: [fix][test] Allow for CLI startup in function integration tests (#26564)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 775373cebf27bfbc0804bf303b0e7389ed7624a4
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Sep 13 18:16:08 2026 +0300

    [fix][test] Allow for CLI startup in function integration tests (#26564)
    
    (cherry picked from commit 376073550aee0fd788b1ad7194c4cdd1677acbcd)
---
 .../pulsar/tests/integration/functions/PulsarFunctionsTest.java     | 3 ++-
 .../apache/pulsar/tests/integration/functions/PulsarStateTest.java  | 6 ++++--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java
index 82418d9d9dc7..68da7de1aa31 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java
@@ -1365,9 +1365,10 @@ public abstract class PulsarFunctionsTest extends PulsarFunctionsTestBase {
 
     private void getFunctionStatus(String functionName, int numMessages, boolean checkRestarts, int parallelism)
             throws Exception {
+        // Each status check starts the admin CLI; allow for startup and command completion on busy CI runners.
         Awaitility.await()
                 .pollInterval(Duration.ofSeconds(1))
-                .atMost(Duration.ofSeconds(15))
+                .atMost(Duration.ofSeconds(30))
                 .ignoreExceptions()
                 .untilAsserted(() ->
                         doGetFunctionStatus(functionName, numMessages, checkRestarts, parallelism));
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarStateTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarStateTest.java
index 9d2510e61030..9d1dc809f374 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarStateTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarStateTest.java
@@ -149,7 +149,8 @@ public abstract class PulsarStateTest extends PulsarStandaloneTestSuite {
         getFunctionInfoNotFound(functionName);
     }
 
-    @Test(groups = {"python_state", "state", "function", "python_function"})
+    // Both passes launch many admin CLI JVMs to verify state operations and cleanup after deletion.
+    @Test(groups = {"python_state", "state", "function", "python_function"}, timeOut = 600_000)
     public void testPythonWordCountFunction() throws Exception {
         if (PulsarMetadataStateStoreProviderImpl.class.getName().equals(stateStoreProvider)) {
             // python function doesn't support metadata state store yet
@@ -163,7 +164,8 @@ public abstract class PulsarStateTest extends PulsarStandaloneTestSuite {
         doTestPythonWordCountFunction(functionName);
     }
 
-    @Test(groups = {"java_state", "state", "function", "java_function"})
+    // Both passes launch many admin CLI JVMs to verify state operations and cleanup after deletion.
+    @Test(groups = {"java_state", "state", "function", "java_function"}, timeOut = 600_000)
     public void testJavaWordCountFunction() throws Exception {
         String functionName = "test-wordcount-java-fn-" + randomName(8);
         doTestJavaWordCountFunction(functionName);


(pulsar) 08/15: [fix][test] Await both DLQ producers in multi-consumer test (#26561)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit e95cbfde259dc14967f7ca9362eb81929722f2a0
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sun Sep 13 18:14:33 2026 +0300

    [fix][test] Await both DLQ producers in multi-consumer test (#26561)
    
    (cherry picked from commit e0ed7749a6b0aecb30d2bf482f328ca54c44cca7)
---
 .../test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java  | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
index dd427cb30dff..844593a7a4b8 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/client/api/DeadLetterTopicTest.java
@@ -1328,11 +1328,10 @@ public class DeadLetterTopicTest extends SharedPulsarBaseTest {
                 .pollInterval(Duration.ofSeconds(1)).untilAsserted(() -> {
             assertTrue(admin.namespaces().getTopics(getNamespace()).contains(deadLetterTopic));
             assertTrue(admin.topics().getSubscriptions(deadLetterTopic).contains(dlqInitialSub));
+            // Each consumer creates its DLQ producer asynchronously after its final redelivery.
+            assertEquals(admin.topics().getStats(deadLetterTopic).getPublishers().size(), 2);
         });
 
-        // We should assert that all consumers are able to produce messages to DLQ
-        assertEquals(admin.topics().getStats(deadLetterTopic).getPublishers().size(), 2);
-
         Consumer<byte[]> deadLetterConsumer = newPulsarClient.newConsumer(Schema.BYTES)
                 .topic(deadLetterTopic)
                 .subscriptionName(dlqInitialSub)


(pulsar) 05/15: [fix][fn] Bound function package downloads from BookKeeper and stop SchedulerManager.close() from hanging on a stuck scheduling round (#26556)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 8f43058b25aab41bcacb40f16cb48a7b86e27e8f
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Sat Sep 12 23:43:02 2026 +0300

    [fix][fn] Bound function package downloads from BookKeeper and stop SchedulerManager.close() from hanging on a stuck scheduling round (#26556)
    
    (cherry picked from commit 0ad91f147bbb1911feaf5b0af9af449882261a39)
---
 .../worker/WorkerUtilsBookKeeperPackageTest.java   | 144 +++++++++++++++++++
 .../pulsar/functions/worker/SchedulerManager.java  |  78 +++++++++--
 .../pulsar/functions/worker/WorkerUtils.java       |   4 +-
 .../functions/worker/dlog/DLInputStream.java       | 152 +++++++++++++--------
 .../functions/worker/SchedulerManagerTest.java     |  52 +++++++
 .../functions/worker/dlog/DLInputStreamTest.java   | 149 +++++++++++++++++---
 6 files changed, 486 insertions(+), 93 deletions(-)

diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/WorkerUtilsBookKeeperPackageTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/WorkerUtilsBookKeeperPackageTest.java
new file mode 100644
index 000000000000..6a62e1d2ec29
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/WorkerUtilsBookKeeperPackageTest.java
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.functions.worker;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.time.Duration;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.distributedlog.api.DistributedLogManager;
+import org.apache.distributedlog.api.namespace.Namespace;
+import org.apache.distributedlog.api.namespace.NamespaceBuilder;
+import org.apache.distributedlog.exceptions.LogNotFoundException;
+import org.apache.pulsar.common.conf.InternalConfigurationData;
+import org.apache.pulsar.functions.worker.dlog.DLInputStream;
+import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * Tests uploading and downloading function packages to and from BookKeeper with the DistributedLog namespace that
+ * the function worker uses.
+ */
+@Test(groups = "functions-worker")
+public class WorkerUtilsBookKeeperPackageTest {
+    private static final Duration READ_TIMEOUT = Duration.ofSeconds(3);
+
+    private LocalBookkeeperEnsemble bkEnsemble;
+    private Namespace dlogNamespace;
+
+    @BeforeClass
+    void setup() throws Exception {
+        bkEnsemble = new LocalBookkeeperEnsemble(1, 0, () -> 0);
+        bkEnsemble.start();
+        InternalConfigurationData internalConf = new InternalConfigurationData(
+                "127.0.0.1:" + bkEnsemble.getZookeeperPort(), null, "/ledgers", null, null);
+        URI dlogUri = WorkerUtils.initializeDlogNamespace(internalConf);
+        WorkerConfig workerConfig = new WorkerConfig();
+        workerConfig.setNumFunctionPackageReplicas(1);
+        dlogNamespace = NamespaceBuilder.newBuilder()
+                .conf(WorkerUtils.getDlogConf(workerConfig))
+                .clientId("function-worker-test")
+                .uri(dlogUri)
+                .build();
+    }
+
+    @AfterClass(alwaysRun = true)
+    void cleanup() throws Exception {
+        if (dlogNamespace != null) {
+            dlogNamespace.close();
+            dlogNamespace = null;
+        }
+        if (bkEnsemble != null) {
+            bkEnsemble.stop();
+            bkEnsemble = null;
+        }
+    }
+
+    private byte[] download(String packagePath) throws IOException {
+        ByteArrayOutputStream out = new ByteArrayOutputStream();
+        try (DistributedLogManager dlm = dlogNamespace.openLog(packagePath);
+             InputStream in = new DLInputStream(dlm, READ_TIMEOUT)) {
+            in.transferTo(out);
+        }
+        return out.toByteArray();
+    }
+
+    @Test
+    public void testUploadAndDownload() throws Exception {
+        byte[] data = new byte[3_000_000];
+        ThreadLocalRandom.current().nextBytes(data);
+        WorkerUtils.uploadToBookKeeper(dlogNamespace, new ByteArrayInputStream(data), "tenant/ns/fn/complete");
+
+        ByteArrayOutputStream out = new ByteArrayOutputStream();
+        WorkerUtils.downloadFromBookkeeper(dlogNamespace, out, "tenant/ns/fn/complete");
+        assertThat(out.toByteArray()).isEqualTo(data);
+    }
+
+    @Test
+    public void testDownloadMissingPackageFails() {
+        assertThatThrownBy(() -> download("tenant/ns/fn/missing"))
+                .isInstanceOf(LogNotFoundException.class);
+    }
+
+    @Test
+    public void testDownloadDeletedPackageFails() throws Exception {
+        WorkerUtils.uploadToBookKeeper(dlogNamespace, new ByteArrayInputStream(new byte[1000]),
+                "tenant/ns/fn/deleted");
+        WorkerUtils.deleteFromBookkeeper(dlogNamespace, "tenant/ns/fn/deleted");
+        assertThatThrownBy(() -> download("tenant/ns/fn/deleted"))
+                .isInstanceOf(LogNotFoundException.class);
+    }
+
+    /**
+     * A log stream that exists but has never been written to has no log segments to read. The download must fail
+     * after the read timeout instead of waiting forever for records to be written.
+     */
+    @Test
+    public void testDownloadEmptyPackageTimesOut() throws Exception {
+        dlogNamespace.createLog("tenant/ns/fn/empty");
+        assertThatThrownBy(() -> download("tenant/ns/fn/empty"))
+                .isInstanceOf(IOException.class)
+                .hasMessageContaining("Timed out");
+    }
+
+    /**
+     * A log stream of an upload that failed before {@link org.apache.pulsar.functions.worker.dlog.DLOutputStream}
+     * wrote the end-of-stream marker. The download must fail after the read timeout instead of waiting forever
+     * for the marker.
+     */
+    @Test
+    public void testDownloadIncompletePackageTimesOut() throws Exception {
+        byte[] data = new byte[1000];
+        try (DistributedLogManager dlm = dlogNamespace.openLog("tenant/ns/fn/incomplete");
+             var writer = dlm.getAppendOnlyStreamWriter()) {
+            writer.write(data);
+            writer.force(false);
+        }
+        assertThatThrownBy(() -> download("tenant/ns/fn/incomplete"))
+                .isInstanceOf(IOException.class)
+                .hasMessageContaining("Timed out");
+    }
+}
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java
index 92c72bef171f..b581c6d7b19f 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java
@@ -117,6 +117,10 @@ public class SchedulerManager implements AutoCloseable {
 
     AtomicBoolean isCompactionNeeded = new AtomicBoolean(false);
     private static final long DEFAULT_ADMIN_API_BACKOFF_SEC = 60;
+    // how long close() waits for an in-progress scheduling round to complete before interrupting it, and then for
+    // the interrupted round to unwind
+    private static final long DEFAULT_CLOSE_SCHEDULING_ROUND_TIMEOUT_MS = TimeUnit.SECONDS.toMillis(10);
+    private volatile long closeSchedulingRoundTimeoutMs = DEFAULT_CLOSE_SCHEDULING_ROUND_TIMEOUT_MS;
     public static final String HEARTBEAT_TENANT = "pulsar-function";
     public static final String HEARTBEAT_NAMESPACE = "heartbeat";
 
@@ -210,8 +214,15 @@ public class SchedulerManager implements AutoCloseable {
                         try {
                             runnable.run();
                         } catch (Throwable th) {
-                            log.error("Encountered error when invoking scheduler [{}]", errMsg);
-                            errorNotifier.triggerError(th);
+                            if (!isRunning) {
+                                // close() interrupts a scheduling round that doesn't complete in time; the
+                                // failure of a round that was cut short by closing is not an error of the worker
+                                log.warn("Scheduler manager was closed while invoking the scheduler [{}]", errMsg,
+                                        th);
+                            } else {
+                                log.error("Encountered error when invoking scheduler [{}]", errMsg, th);
+                                errorNotifier.triggerError(th);
+                            }
                         }
                     }
                 } finally {
@@ -738,33 +749,70 @@ public class SchedulerManager implements AutoCloseable {
     }
 
     @Override
-    public synchronized void close() {
+    public void close() {
         log.info("Closing scheduler manager");
-        // make sure we are not closing while a scheduling is being calculated
-        schedulerLock.lock();
-        try {
+        final ThreadPoolExecutor executor;
+        final ScheduledExecutorService scheduledExecutor;
+        final Producer<byte[]> producer;
+        // don't hold the monitor while waiting for the scheduler lock: a scheduling round holds the scheduler lock
+        // while calling synchronized methods of this class
+        synchronized (this) {
             isRunning = false;
+            executor = executorService;
+            scheduledExecutor = scheduledExecutorService;
+            producer = exclusiveProducer;
+        }
 
-            if (scheduledExecutorService != null) {
-                scheduledExecutorService.shutdown();
-            }
+        if (scheduledExecutor != null) {
+            scheduledExecutor.shutdown();
+        }
+        if (executor != null) {
+            // stop accepting new scheduling rounds, a round that is already running keeps running
+            executor.shutdown();
+        }
 
-            if (executorService != null) {
-                executorService.shutdown();
+        // make sure we are not closing the assignment producer while a scheduling round is being calculated.
+        // A round can block for a long time (for example while downloading a function package), so don't wait for
+        // it indefinitely: interrupt it and continue closing.
+        boolean locked = tryLockSchedulerLock();
+        if (!locked && executor != null) {
+            log.warn("Scheduling round did not complete in {} ms, interrupting it", closeSchedulingRoundTimeoutMs);
+            executor.shutdownNow();
+            locked = tryLockSchedulerLock();
+            if (!locked) {
+                log.warn("Interrupted scheduling round did not complete in {} ms, closing anyway",
+                        closeSchedulingRoundTimeoutMs);
             }
-
-            if (exclusiveProducer != null) {
+        }
+        try {
+            if (producer != null) {
                 try {
-                    exclusiveProducer.close();
+                    producer.close();
                 } catch (PulsarClientException e) {
                     log.warn("Failed to shutdown scheduler manager assignment producer", e);
                 }
             }
         } finally {
-            schedulerLock.unlock();
+            if (locked) {
+                schedulerLock.unlock();
+            }
+        }
+    }
+
+    private boolean tryLockSchedulerLock() {
+        try {
+            return schedulerLock.tryLock(closeSchedulingRoundTimeoutMs, TimeUnit.MILLISECONDS);
+        } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+            return false;
         }
     }
 
+    @VisibleForTesting
+    void setCloseSchedulingRoundTimeoutMs(long closeSchedulingRoundTimeoutMs) {
+        this.closeSchedulingRoundTimeoutMs = closeSchedulingRoundTimeoutMs;
+    }
+
     static String checkHeartBeatFunction(Instance funInstance) {
         if (funInstance.getFunctionMetaData() != null
                 && funInstance.getFunctionMetaData().getFunctionDetails() != null) {
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerUtils.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerUtils.java
index c86c6e9a193c..eccbb93c23ad 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerUtils.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerUtils.java
@@ -121,8 +121,8 @@ public final class WorkerUtils {
                                               OutputStream outputStream,
                                               String packagePath) throws IOException {
         log.info("Downloading {} from BK...", packagePath);
-        DistributedLogManager dlm = namespace.openLog(packagePath);
-        try (InputStream in = new DLInputStream(dlm)) {
+        try (DistributedLogManager dlm = namespace.openLog(packagePath);
+             InputStream in = new DLInputStream(dlm)) {
             int read = 0;
             byte[] bytes = new byte[1024];
             while ((read = in.read(bytes)) != -1) {
diff --git a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLInputStream.java b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLInputStream.java
index f009f57d1aed..2f9d063e9ced 100644
--- a/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLInputStream.java
+++ b/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLInputStream.java
@@ -20,35 +20,47 @@ package org.apache.pulsar.functions.worker.dlog;
 
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.InterruptedIOException;
+import java.time.Duration;
+import java.util.ArrayDeque;
+import java.util.Deque;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import org.apache.distributedlog.DLSN;
 import org.apache.distributedlog.LogRecordWithDLSN;
+import org.apache.distributedlog.api.AsyncLogReader;
 import org.apache.distributedlog.api.DistributedLogManager;
-import org.apache.distributedlog.api.LogReader;
 import org.apache.distributedlog.exceptions.EndOfStreamException;
 
 /**
  * DistributedLog Input Stream.
+ *
+ * <p>Reads the records of a log stream until the end-of-stream marker written by {@link DLOutputStream#close()}.
+ * A log stream is a tailing log: a reader of a stream that has no end-of-stream marker (for example because the
+ * writer failed before completing the upload, or the stream was never written to) waits for more records to arrive
+ * forever. To avoid blocking the calling thread indefinitely, waiting for the next records is bounded by a read
+ * timeout and fails with an {@link IOException} when it expires.
  */
 public class DLInputStream extends InputStream {
 
-  private LogRecordWithInputStream currentLogRecord = null;
-  private final DistributedLogManager dlm;
-  private LogReader reader;
-  private boolean eos = false;
-
-  // Cache the input stream for a log record.
-  private static class LogRecordWithInputStream {
-    private final InputStream payloadStream;
-
-    LogRecordWithInputStream(LogRecordWithDLSN logRecord) {
-      this.payloadStream = logRecord.getPayLoadInputStream();
-    }
+  /**
+   * Default maximum time to wait for the next records of the log stream to become available.
+   */
+  public static final Duration DEFAULT_READ_TIMEOUT = Duration.ofSeconds(60);
 
-    InputStream getPayLoadInputStream() {
-      return payloadStream;
-    }
+  private static final int READ_BATCH_SIZE = 100;
 
-  }
+  private final DistributedLogManager dlm;
+  private final AsyncLogReader reader;
+  private final long readTimeoutMs;
+  private final Deque<LogRecordWithDLSN> pendingRecords = new ArrayDeque<>();
+  // payload of the log record that is currently being consumed
+  private InputStream currentPayload;
+  private boolean eos = false;
 
   /**
    * Construct DistributedLog input stream.
@@ -56,38 +68,73 @@ public class DLInputStream extends InputStream {
    * @param dlm the Distributed Log Manager to access the stream
    */
   public DLInputStream(DistributedLogManager dlm) throws IOException {
-    this.dlm = dlm;
-    reader = dlm.getInputStream(DLSN.InitialDLSN);
+    this(dlm, DEFAULT_READ_TIMEOUT);
   }
 
   /**
-   * Get input stream representing next entry in the
-   * ledger.
+   * Construct DistributedLog input stream.
    *
-   * @return input stream, or null if no more entries
+   * @param dlm the Distributed Log Manager to access the stream
+   * @param readTimeout maximum time to wait for the next records of the stream to become available
    */
-  private LogRecordWithInputStream nextLogRecord() throws IOException {
+  public DLInputStream(DistributedLogManager dlm, Duration readTimeout) throws IOException {
+    this.dlm = dlm;
+    this.readTimeoutMs = readTimeout.toMillis();
+    this.reader = await(dlm.openAsyncLogReader(DLSN.InitialDLSN), "open a reader for");
+  }
+
+  private <T> T await(CompletableFuture<T> future, String operation) throws IOException {
     try {
-      return nextLogRecord(reader);
-    } catch (EndOfStreamException e) {
-      eos = true;
-      return null;
+      return future.get(readTimeoutMs, TimeUnit.MILLISECONDS);
+    } catch (TimeoutException e) {
+      future.cancel(false);
+      throw new IOException("Timed out after " + readTimeoutMs + " ms waiting to " + operation + " log stream "
+          + dlm.getStreamName(), e);
+    } catch (InterruptedException e) {
+      Thread.currentThread().interrupt();
+      throw new InterruptedIOException("Interrupted while waiting to " + operation + " log stream "
+          + dlm.getStreamName());
+    } catch (ExecutionException e) {
+      Throwable cause = e.getCause();
+      if (cause instanceof IOException) {
+        throw (IOException) cause;
+      }
+      throw new IOException("Failed to " + operation + " log stream " + dlm.getStreamName(), cause);
     }
   }
 
-  private static LogRecordWithInputStream nextLogRecord(LogReader reader) throws IOException {
-    LogRecordWithDLSN record = reader.readNext(false);
+  /**
+   * Get the next log record of the stream.
+   *
+   * @return the next log record, or null when the end of the stream has been reached
+   */
+  private LogRecordWithDLSN nextLogRecord() throws IOException {
+    if (pendingRecords.isEmpty() && !eos) {
+      try {
+        List<LogRecordWithDLSN> records = await(reader.readBulk(READ_BATCH_SIZE), "read");
+        pendingRecords.addAll(records);
+      } catch (EndOfStreamException e) {
+        eos = true;
+      }
+    }
+    return pendingRecords.pollFirst();
+  }
 
-    if (null != record) {
-      return new LogRecordWithInputStream(record);
-    } else {
-      record = reader.readNext(false);
-      if (null != record) {
-        return new LogRecordWithInputStream(record);
-      } else {
+  /**
+   * Get the payload of the log record that is currently being consumed, moving on to the next log record when the
+   * current payload has been fully consumed.
+   *
+   * @return the payload input stream, or null when the end of the stream has been reached
+   */
+  private InputStream currentPayload() throws IOException {
+    if (currentPayload == null) {
+      LogRecordWithDLSN record = nextLogRecord();
+      if (record == null) {
         return null;
       }
+      currentPayload = record.getPayLoadInputStream();
     }
+    return currentPayload;
   }
 
   @Override
@@ -102,35 +149,32 @@ public class DLInputStream extends InputStream {
 
   @Override
   public int read(byte[] b, int off, int len) throws IOException {
-    if (eos) {
-      return -1;
+    Objects.checkFromIndexSize(off, len, b.length);
+    if (len == 0) {
+      return 0;
     }
-
     int read = 0;
-    if (currentLogRecord == null) {
-      currentLogRecord = nextLogRecord();
-      if (currentLogRecord == null) {
-        return read;
-      }
-    }
-
     while (read < len) {
-      int thisread = currentLogRecord.getPayLoadInputStream().read(b, off + read, len - read);
+      InputStream payload = currentPayload();
+      if (payload == null) {
+        break;
+      }
+      int thisread = payload.read(b, off + read, len - read);
       if (thisread == -1) {
-        currentLogRecord = nextLogRecord();
-        if (currentLogRecord == null) {
-          return read;
-        }
+        currentPayload = null;
       } else {
         read += thisread;
       }
     }
-    return read;
+    return read == 0 ? -1 : read;
   }
 
   @Override
   public void close() throws IOException {
-    reader.close();
-    dlm.close();
+    try {
+      await(reader.asyncClose(), "close the reader of");
+    } finally {
+      dlm.close();
+    }
   }
 }
diff --git a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
index 9e7b8d4fa6b1..87d4e82450d3 100644
--- a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
+++ b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
@@ -23,6 +23,7 @@ import static org.mockito.Mockito.any;
 import static org.mockito.Mockito.anyBoolean;
 import static org.mockito.Mockito.anyInt;
 import static org.mockito.Mockito.anyString;
+import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.doReturn;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.spy;
@@ -41,12 +42,14 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
 import lombok.extern.slf4j.Slf4j;
 import lombok.val;
 import org.apache.pulsar.client.admin.LongRunningProcessStatus;
@@ -1103,6 +1106,55 @@ public class SchedulerManagerTest {
         }
     }
 
+    @Test(timeOut = 60000)
+    public void testCloseInterruptsStuckSchedulingRound() throws Exception {
+        List<Function.FunctionMetaData> functionMetaDataList = new LinkedList<>();
+        Function.FunctionMetaData function1 = Function.FunctionMetaData.newBuilder()
+                .setFunctionDetails(Function.FunctionDetails.newBuilder().setName("func-1")
+                        .setNamespace("namespace-1").setTenant("tenant-1").setParallelism(1)).setVersion(1)
+                .build();
+        functionMetaDataList.add(function1);
+        doReturn(functionMetaDataList).when(functionMetaDataManager).getAllFunctionMetaData();
+        ThreadRuntimeFactory factory = mock(ThreadRuntimeFactory.class);
+        doReturn(factory).when(functionRuntimeManager).getRuntimeFactory();
+        doReturn(new HashMap<>()).when(functionRuntimeManager).getCurrentAssignments();
+        List<WorkerInfo> workerInfoList = new LinkedList<>();
+        workerInfoList.add(WorkerInfo.of("worker-1", "workerHostname-1", 5000));
+        doReturn(workerInfoList).when(membershipManager).getCurrentMembership();
+        doReturn(true).when(leaderService).isLeader();
+
+        // processing the new assignment blocks until interrupted, like a function package download that never
+        // completes
+        CountDownLatch roundStuck = new CountDownLatch(1);
+        AtomicBoolean roundInterrupted = new AtomicBoolean();
+        doAnswer(invocation -> {
+            roundStuck.countDown();
+            try {
+                new CountDownLatch(1).await();
+            } catch (InterruptedException e) {
+                roundInterrupted.set(true);
+                throw new RuntimeException(e);
+            }
+            return null;
+        }).when(functionRuntimeManager).processAssignment(any(Assignment.class));
+
+        schedulerManager.setCloseSchedulingRoundTimeoutMs(500);
+        schedulerManager.initialize(schedulerManager.acquireExclusiveWrite(() -> true));
+        schedulerManager.schedule();
+        assertTrue(roundStuck.await(30, TimeUnit.SECONDS));
+
+        long closeStart = System.nanoTime();
+        schedulerManager.close();
+        long closeDurationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - closeStart);
+        assertTrue(closeDurationMs < 10000, "close() took " + closeDurationMs + " ms");
+        assertTrue(roundInterrupted.get());
+        verify(producer, times(1)).close();
+        // the failure of the interrupted scheduling round is not a worker error
+        verify(errorNotifier, times(0)).triggerError(any());
+        assertTrue(schedulerManager.getSchedulerLock().tryLock(30, TimeUnit.SECONDS));
+        schedulerManager.getSchedulerLock().unlock();
+    }
+
     private void callSchedule() throws InterruptedException,
             TimeoutException, ExecutionException, WorkerUtils.NotLeaderAnymore {
 
diff --git a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLInputStreamTest.java b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLInputStreamTest.java
index 60b01391aeae..98058f773f31 100644
--- a/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLInputStreamTest.java
+++ b/pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLInputStreamTest.java
@@ -18,8 +18,10 @@
  */
 package org.apache.pulsar.functions.worker.dlog;
 
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 import static org.mockito.Mockito.any;
-import static org.mockito.Mockito.anyBoolean;
+import static org.mockito.Mockito.anyInt;
 import static org.mockito.Mockito.eq;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.times;
@@ -27,12 +29,19 @@ import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
 import static org.testng.AssertJUnit.assertEquals;
 import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
+import java.time.Duration;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeoutException;
 import org.apache.distributedlog.DLSN;
 import org.apache.distributedlog.LogRecordWithDLSN;
+import org.apache.distributedlog.api.AsyncLogReader;
 import org.apache.distributedlog.api.DistributedLogManager;
-import org.apache.distributedlog.api.LogReader;
 import org.apache.distributedlog.exceptions.EndOfStreamException;
+import org.apache.distributedlog.exceptions.LogNotFoundException;
 import org.testng.annotations.Test;
 
 /**
@@ -40,22 +49,44 @@ import org.testng.annotations.Test;
  */
 public class DLInputStreamTest {
 
+    private static DistributedLogManager mockDlm(AsyncLogReader reader) {
+        DistributedLogManager dlm = mock(DistributedLogManager.class);
+        when(dlm.getStreamName()).thenReturn("test-stream");
+        when(dlm.openAsyncLogReader(any(DLSN.class))).thenReturn(CompletableFuture.completedFuture(reader));
+        return dlm;
+    }
+
+    private static AsyncLogReader mockReader() {
+        AsyncLogReader reader = mock(AsyncLogReader.class);
+        when(reader.asyncClose()).thenReturn(CompletableFuture.completedFuture(null));
+        return reader;
+    }
+
+    private static LogRecordWithDLSN mockRecord(byte[] payload) {
+        LogRecordWithDLSN record = mock(LogRecordWithDLSN.class);
+        when(record.getPayLoadInputStream()).thenReturn(new ByteArrayInputStream(payload));
+        return record;
+    }
+
     /**
      * Test Case: reader hits eos (end of stream).
      */
     @Test
     public void testReadEos() throws Exception {
-        DistributedLogManager dlm = mock(DistributedLogManager.class);
-        LogReader reader = mock(LogReader.class);
-        when(dlm.getInputStream(any(DLSN.class))).thenReturn(reader);
-        when(reader.readNext(anyBoolean())).thenThrow(new EndOfStreamException("eos"));
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
+        when(reader.readBulk(anyInt())).thenReturn(CompletableFuture.failedFuture(new EndOfStreamException("eos")));
 
         byte[] b = new byte[1];
         DLInputStream in = new DLInputStream(dlm);
-        assertEquals("Should return 0 when reading an empty eos stream",
-            0, in.read(b, 0, 1));
         assertEquals("Should return -1 when reading an empty eos stream",
             -1, in.read(b, 0, 1));
+        assertEquals("Should keep returning -1 when reading an empty eos stream",
+            -1, in.read(b, 0, 1));
+        assertEquals("Should return -1 when reading a byte from an empty eos stream",
+            -1, in.read());
+        // the end of stream is reached only once
+        verify(reader, times(1)).readBulk(anyInt());
     }
 
     /**
@@ -63,15 +94,14 @@ public class DLInputStreamTest {
      */
     @Test
     public void testClose() throws Exception {
-        DistributedLogManager dlm = mock(DistributedLogManager.class);
-        LogReader reader = mock(LogReader.class);
-        when(dlm.getInputStream(any(DLSN.class))).thenReturn(reader);
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
 
         DLInputStream in = new DLInputStream(dlm);
-        verify(dlm, times(1)).getInputStream(eq(DLSN.InitialDLSN));
+        verify(dlm, times(1)).openAsyncLogReader(eq(DLSN.InitialDLSN));
         in.close();
+        verify(reader, times(1)).asyncClose();
         verify(dlm, times(1)).close();
-        verify(reader, times(1)).close();
     }
 
     /**
@@ -79,17 +109,14 @@ public class DLInputStreamTest {
      */
     @Test
     public void testRead() throws Exception {
-        DistributedLogManager dlm = mock(DistributedLogManager.class);
-        LogReader reader = mock(LogReader.class);
-        when(dlm.getInputStream(any(DLSN.class))).thenReturn(reader);
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
 
         byte[] data = "test-read".getBytes(StandardCharsets.UTF_8);
-        LogRecordWithDLSN record = mock(LogRecordWithDLSN.class);
-        when(record.getPayLoadInputStream())
-            .thenReturn(new ByteArrayInputStream(data));
-        when(reader.readNext(anyBoolean()))
-            .thenReturn(record)
-            .thenThrow(new EndOfStreamException("eos"));
+        LogRecordWithDLSN record = mockRecord(data);
+        when(reader.readBulk(anyInt()))
+            .thenReturn(CompletableFuture.completedFuture(List.of(record)))
+            .thenReturn(CompletableFuture.failedFuture(new EndOfStreamException("eos")));
 
         DLInputStream in = new DLInputStream(dlm);
         int numReads = 0;
@@ -101,4 +128,82 @@ public class DLInputStreamTest {
         assertEquals(data.length, numReads);
     }
 
+    /**
+     * Test Case: a read spans multiple records and batches of records.
+     */
+    @Test
+    public void testReadAcrossRecords() throws Exception {
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
+
+        List<LogRecordWithDLSN> firstBatch = List.of(mockRecord("ab".getBytes(StandardCharsets.UTF_8)),
+            mockRecord("cd".getBytes(StandardCharsets.UTF_8)));
+        List<LogRecordWithDLSN> secondBatch = List.of(mockRecord("ef".getBytes(StandardCharsets.UTF_8)));
+        when(reader.readBulk(anyInt()))
+            .thenReturn(CompletableFuture.completedFuture(firstBatch))
+            .thenReturn(CompletableFuture.completedFuture(secondBatch))
+            .thenReturn(CompletableFuture.failedFuture(new EndOfStreamException("eos")));
+
+        try (InputStream in = new DLInputStream(dlm)) {
+            byte[] b = new byte[5];
+            assertThat(in.read(b, 0, 5)).isEqualTo(5);
+            assertThat(new String(b, StandardCharsets.UTF_8)).isEqualTo("abcde");
+            assertThat(in.read(b, 0, 5)).isEqualTo(1);
+            assertThat(b[0]).isEqualTo((byte) 'f');
+            assertThat(in.read(b, 0, 5)).isEqualTo(-1);
+        }
+    }
+
+    /**
+     * Test Case: the stream never delivers the next records (for example an incomplete upload without an
+     * end-of-stream marker). The read must fail after the read timeout instead of blocking forever.
+     */
+    @Test
+    public void testReadTimesOut() throws Exception {
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
+        CompletableFuture<List<LogRecordWithDLSN>> neverCompleted = new CompletableFuture<>();
+        when(reader.readBulk(anyInt())).thenReturn(neverCompleted);
+
+        try (InputStream in = new DLInputStream(dlm, Duration.ofMillis(200))) {
+            assertThatThrownBy(() -> in.read(new byte[1], 0, 1))
+                .isInstanceOf(IOException.class)
+                .hasMessageContaining("Timed out")
+                .hasMessageContaining("test-stream")
+                .hasCauseInstanceOf(TimeoutException.class);
+        }
+        assertThat(neverCompleted).isCancelled();
+    }
+
+    /**
+     * Test Case: reading fails with the failure of the log stream read.
+     */
+    @Test
+    public void testReadFailure() throws Exception {
+        AsyncLogReader reader = mockReader();
+        DistributedLogManager dlm = mockDlm(reader);
+        when(reader.readBulk(anyInt()))
+            .thenReturn(CompletableFuture.failedFuture(new LogNotFoundException("deleted")));
+
+        try (InputStream in = new DLInputStream(dlm)) {
+            assertThatThrownBy(() -> in.read(new byte[1], 0, 1))
+                .isInstanceOf(LogNotFoundException.class)
+                .hasMessage("deleted");
+        }
+    }
+
+    /**
+     * Test Case: opening the reader fails.
+     */
+    @Test
+    public void testOpenReaderFailure() {
+        DistributedLogManager dlm = mock(DistributedLogManager.class);
+        when(dlm.getStreamName()).thenReturn("test-stream");
+        when(dlm.openAsyncLogReader(any(DLSN.class)))
+            .thenReturn(CompletableFuture.failedFuture(new LogNotFoundException("missing")));
+
+        assertThatThrownBy(() -> new DLInputStream(dlm))
+            .isInstanceOf(LogNotFoundException.class)
+            .hasMessage("missing");
+    }
 }


(pulsar) 01/15: [fix][broker] Send ActiveConsumerChange for non-persistent Failover subscriptions (#26482)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 63b4721a62ad2b243db46d588fc6bb2c3a0a0fb1
Author: Dream95 <86...@qq.com>
AuthorDate: Sun Sep 13 02:46:31 2026 +0800

    [fix][broker] Send ActiveConsumerChange for non-persistent Failover subscriptions (#26482)
    
    Signed-off-by: Dream95 <zh...@163.com>
    (cherry picked from commit 815833961e76d351494d7cc379dd8ee52d125c0d)
---
 ...onPersistentDispatcherSingleActiveConsumer.java |   4 +-
 ...onPersistentDispatcherFailoverConsumerTest.java | 247 +++++++++++++++++++++
 2 files changed, 250 insertions(+), 1 deletion(-)

diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherSingleActiveConsumer.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherSingleActiveConsumer.java
index 26bb5a791cfd..a8d9790d96b0 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherSingleActiveConsumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherSingleActiveConsumer.java
@@ -101,7 +101,9 @@ public final class NonPersistentDispatcherSingleActiveConsumer extends AbstractD
 
     @Override
     protected void scheduleReadOnActiveConsumer() {
-        // No-op
+        // Non-persistent topics have no cursor to rewind/read, but Failover consumers still need
+        // ActiveConsumerChange notifications (same as PersistentDispatcherSingleActiveConsumer).
+        notifyActiveConsumerChanged(getActiveConsumer());
     }
 
     @Override
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherFailoverConsumerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherFailoverConsumerTest.java
new file mode 100644
index 000000000000..f2ae4817c0c2
--- /dev/null
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherFailoverConsumerTest.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service.nonpersistent;
+
+import static org.apache.pulsar.common.protocol.Commands.DEFAULT_CONSUMER_EPOCH;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertTrue;
+import io.netty.buffer.ByteBuf;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelHandlerContext;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import org.apache.pulsar.broker.ServiceConfiguration;
+import org.apache.pulsar.broker.namespace.NamespaceService;
+import org.apache.pulsar.broker.service.Consumer;
+import org.apache.pulsar.broker.service.PulsarCommandSenderImpl;
+import org.apache.pulsar.broker.service.ServerCnx;
+import org.apache.pulsar.broker.testcontext.PulsarTestContext;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.common.api.proto.BaseCommand;
+import org.apache.pulsar.common.api.proto.CommandActiveConsumerChange;
+import org.apache.pulsar.common.api.proto.CommandSubscribe.SubType;
+import org.apache.pulsar.common.api.proto.ProtocolVersion;
+import org.apache.pulsar.common.naming.NamespaceBundle;
+import org.apache.pulsar.common.semaphore.AsyncDualMemoryLimiter;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+/**
+ * Verifies non-persistent Failover subscriptions notify clients of active-consumer changes.
+ */
+@Test(groups = "broker")
+public class NonPersistentDispatcherFailoverConsumerTest {
+
+    private static final String TOPIC_NAME = "non-persistent://prop/ns-abc/failover-notify";
+
+    private record ActiveChange(long consumerId, boolean isActive) {
+    }
+
+    private record TestDispatcher(NonPersistentTopic topic, NonPersistentSubscription sub,
+                                  NonPersistentDispatcherSingleActiveConsumer dispatcher) {
+    }
+
+    private PulsarTestContext pulsarTestContext;
+    private ServerCnx serverCnx;
+    private LinkedBlockingQueue<ActiveChange> consumerChanges;
+
+    @BeforeMethod
+    public void setup() throws Exception {
+        ServiceConfiguration svcConfig = new ServiceConfiguration();
+        svcConfig.setBrokerShutdownTimeoutMs(0L);
+        svcConfig.setLoadBalancerOverrideBrokerNicSpeedGbps(Optional.of(1.0d));
+        svcConfig.setClusterName("pulsar-cluster");
+        svcConfig.setSystemTopicEnabled(false);
+        svcConfig.setTopicLevelPoliciesEnabled(false);
+        svcConfig.setActiveConsumerFailoverDelayTimeMillis(0);
+        pulsarTestContext = PulsarTestContext.builderForNonStartableContext()
+                .config(svcConfig)
+                .spyByDefault()
+                .build();
+
+        consumerChanges = new LinkedBlockingQueue<>();
+        ChannelHandlerContext channelCtx = mock(ChannelHandlerContext.class);
+        Channel channel = mock(Channel.class);
+        when(channelCtx.channel()).thenReturn(channel);
+        doAnswer(invocationOnMock -> {
+            ByteBuf buf = invocationOnMock.getArgument(0);
+            ByteBuf cmdBuf = buf.retainedSlice(4, buf.writerIndex() - 4);
+            try {
+                int cmdSize = (int) cmdBuf.readUnsignedInt();
+                BaseCommand cmd = new BaseCommand();
+                cmd.parseFrom(cmdBuf, cmdSize);
+                if (cmd.hasActiveConsumerChange()) {
+                    CommandActiveConsumerChange change = cmd.getActiveConsumerChange();
+                    consumerChanges.put(new ActiveChange(change.getConsumerId(), change.isIsActive()));
+                }
+            } finally {
+                cmdBuf.release();
+                buf.release();
+            }
+            return null;
+        }).when(channelCtx).writeAndFlush(any(), any());
+
+        AsyncDualMemoryLimiter maxTopicListInFlightLimiter = mock(AsyncDualMemoryLimiter.class);
+        serverCnx = createServerCnx(channelCtx, maxTopicListInFlightLimiter);
+
+        NamespaceService nsSvc = pulsarTestContext.getPulsarService().getNamespaceService();
+        doReturn(true).when(nsSvc).isServiceUnitOwned(any(NamespaceBundle.class));
+        doReturn(CompletableFuture.completedFuture(mock(NamespaceBundle.class))).when(nsSvc).getBundleAsync(any());
+        doReturn(CompletableFuture.completedFuture(true)).when(nsSvc).checkBundleOwnership(any(), any());
+    }
+
+    @AfterMethod(alwaysRun = true)
+    public void shutdown() throws Exception {
+        if (pulsarTestContext != null) {
+            pulsarTestContext.close();
+            pulsarTestContext = null;
+        }
+    }
+
+    private ServerCnx createServerCnx(ChannelHandlerContext channelCtx,
+                                      AsyncDualMemoryLimiter maxTopicListInFlightLimiter) {
+        ServerCnx cnx = pulsarTestContext.createServerCnxSpy();
+        doReturn(true).when(cnx).isActive();
+        doReturn(true).when(cnx).isWritable();
+        doReturn(new InetSocketAddress("localhost", 1234)).when(cnx).clientAddress();
+        when(cnx.getRemoteEndpointProtocolVersion()).thenReturn(ProtocolVersion.v12.getValue());
+        when(cnx.ctx()).thenReturn(channelCtx);
+        doReturn(CompletableFuture.completedFuture(Optional.of(true))).when(cnx).checkConnectionLiveness();
+        doReturn(new PulsarCommandSenderImpl(null, cnx, maxTopicListInFlightLimiter)).when(cnx).getCommandSender();
+        return cnx;
+    }
+
+    private TestDispatcher newDispatcher(SubType subType) {
+        NonPersistentTopic topic = new NonPersistentTopic(TOPIC_NAME, pulsarTestContext.getBrokerService());
+        NonPersistentSubscription sub = new NonPersistentSubscription(topic, "sub-1", Collections.emptyMap());
+        return new TestDispatcher(topic, sub,
+                new NonPersistentDispatcherSingleActiveConsumer(subType, 0, topic, sub));
+    }
+
+    private Consumer newConsumer(TestDispatcher td, SubType subType, long consumerId, String consumerName) {
+        return newConsumer(td, subType, consumerId, 0, consumerName, serverCnx);
+    }
+
+    private Consumer newConsumer(TestDispatcher td, SubType subType, long consumerId, int priorityLevel,
+                                 String consumerName, ServerCnx cnx) {
+        return new Consumer(td.sub(), subType, td.topic().getName(), consumerId, priorityLevel, consumerName,
+                true, cnx, "myrole-1", Collections.emptyMap(), false, null, MessageId.latest, DEFAULT_CONSUMER_EPOCH);
+    }
+
+    private List<ActiveChange> takeExactChanges(int expected) {
+        List<ActiveChange> actual = new ArrayList<>();
+        consumerChanges.drainTo(actual);
+        assertEquals(actual.size(), expected, "unexpected ActiveConsumerChange notifications: " + actual);
+        return actual;
+    }
+
+    private void assertNoActiveConsumerChanges() {
+        assertTrue(consumerChanges.isEmpty(),
+                "unexpected ActiveConsumerChange notifications: " + consumerChanges);
+    }
+
+    @Test(timeOut = 10000)
+    public void testActiveConsumerChangeNotifications() throws Exception {
+        TestDispatcher td = newDispatcher(SubType.Failover);
+        NonPersistentDispatcherSingleActiveConsumer dispatcher = td.dispatcher();
+        assertFalse(dispatcher.isConsumerConnected());
+
+        Consumer consumer1 = newConsumer(td, SubType.Failover, 1, "Cons1");
+        dispatcher.addConsumer(consumer1).get();
+        assertSame(dispatcher.getActiveConsumer(), consumer1);
+        assertEquals(takeExactChanges(1), List.of(new ActiveChange(1, true)));
+
+        Consumer consumer2 = newConsumer(td, SubType.Failover, 2, "Cons2");
+        dispatcher.addConsumer(consumer2).get();
+        assertSame(dispatcher.getActiveConsumer(), consumer1);
+        assertEquals(takeExactChanges(1), List.of(new ActiveChange(2, false)));
+
+        Consumer consumer0 = newConsumer(td, SubType.Failover, 0, "Cons0");
+        dispatcher.addConsumer(consumer0).get();
+        assertSame(dispatcher.getActiveConsumer(), consumer0);
+        // consumers is sorted by name before notify, so Cons0, Cons1, Cons2.
+        assertEquals(takeExactChanges(3), List.of(
+                new ActiveChange(0, true),
+                new ActiveChange(1, false),
+                new ActiveChange(2, false)));
+
+        dispatcher.removeConsumer(consumer0);
+        assertSame(dispatcher.getActiveConsumer(), consumer1);
+        assertEquals(takeExactChanges(2), List.of(
+                new ActiveChange(1, true),
+                new ActiveChange(2, false)));
+
+        dispatcher.removeConsumer(consumer2);
+        assertSame(dispatcher.getActiveConsumer(), consumer1);
+        assertNoActiveConsumerChanges();
+
+        dispatcher.removeConsumer(consumer1);
+        assertFalse(dispatcher.isConsumerConnected());
+        assertNoActiveConsumerChanges();
+    }
+
+    @Test(timeOut = 10000)
+    public void testAddConsumerWhenClosed() throws Exception {
+        NonPersistentDispatcherSingleActiveConsumer dispatcher = newDispatcher(SubType.Failover).dispatcher();
+        dispatcher.close().get();
+
+        Consumer consumer = mock(Consumer.class);
+        dispatcher.addConsumer(consumer).get();
+        verify(consumer, times(1)).disconnect();
+        assertEquals(dispatcher.getConsumers().size(), 0);
+        assertNoActiveConsumerChanges();
+    }
+
+    @Test(timeOut = 10000)
+    public void testHigherPriorityConsumerBecomesActive() throws Exception {
+        TestDispatcher td = newDispatcher(SubType.Failover);
+        NonPersistentDispatcherSingleActiveConsumer dispatcher = td.dispatcher();
+
+        Consumer consumer1 = newConsumer(td, SubType.Failover, 1, 1, "Cons1", serverCnx);
+        dispatcher.addConsumer(consumer1).get();
+        assertEquals(takeExactChanges(1), List.of(new ActiveChange(1, true)));
+
+        Consumer consumer2 = newConsumer(td, SubType.Failover, 2, 1, "Cons2", serverCnx);
+        dispatcher.addConsumer(consumer2).get();
+        assertEquals(takeExactChanges(1), List.of(new ActiveChange(2, false)));
+
+        Consumer consumer3 = newConsumer(td, SubType.Failover, 3, 0, "Cons3", serverCnx);
+        dispatcher.addConsumer(consumer3).get();
+        assertSame(dispatcher.getActiveConsumer(), consumer3);
+        assertEquals(takeExactChanges(3), List.of(
+                new ActiveChange(3, true),
+                new ActiveChange(1, false),
+                new ActiveChange(2, false)));
+    }
+}


(pulsar) 13/15: [fix] Fix cursor deadlock when skipping non-recoverable entries during mark-delete persistence (#26570)

Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

lhotari pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git

commit 2766ade8c41c353f3745ab4a5696979971fc19cb
Author: void-ptr974 <vo...@protonmail.com>
AuthorDate: Mon Sep 14 08:43:11 2026 +0800

    [fix] Fix cursor deadlock when skipping non-recoverable entries during mark-delete persistence (#26570)
    
    (cherry picked from commit f613b16fe4a4507cc61f7d26c552e42b6da7eef2)
---
 .../bookkeeper/mledger/impl/ManagedCursorImpl.java | 37 +++++++++------------
 .../bookkeeper/mledger/impl/ManagedCursorTest.java | 38 ++++++++++++++++++++++
 2 files changed, 53 insertions(+), 22 deletions(-)

diff --git a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
index 7244f1865cc3..501487366d29 100644
--- a/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
+++ b/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
@@ -3170,7 +3170,7 @@ public class ManagedCursorImpl implements ManagedCursor {
      * Manually acknowledge all entries from startPosition to endPosition.
      * - Since this is an uncommon event, we focus on maintainability. So we do not modify
      *   {@link #individualDeletedMessages} and {@link #batchDeletedIndexes}, but call
-     *   {@link #asyncDelete(Position, AsyncCallbacks.DeleteCallback, Object)}.
+     *   {@link #asyncDelete(Iterable, AsyncCallbacks.DeleteCallback, Object)}.
      * - This method is valid regardless of the consumer ACK type.
      * - If there is a consumer ack request after this event, it will also work.
      */
@@ -3187,31 +3187,24 @@ public class ManagedCursorImpl implements ManagedCursor {
             return;
         }
 
-        lock.writeLock().lock();
         log.warn("[{}] [{}] Since these entry for ledger [{}] is lost and the autoSkipNonRecoverableData is true, "
                         + "these entries [{}:{}) will be auto acknowledge in subscription",
                 ledger.getName(), name, ledgerId, startEntryId, endEntryId);
-        try {
-            for (long i = startEntryId; i < endEntryId; i++) {
-                if (!individualDeletedMessages.contains(ledgerId, i)) {
-                    asyncDelete(PositionFactory.create(ledgerId, i), new AsyncCallbacks.DeleteCallback() {
-                        @Override
-                        public void deleteComplete(Object ctx) {
-                            // ignore.
-                        }
+        asyncDelete(() -> LongStream.range(startEntryId, endEntryId)
+                        .mapToObj(i -> PositionFactory.create(ledgerId, i)).iterator(),
+                new AsyncCallbacks.DeleteCallback() {
+                    @Override
+                    public void deleteComplete(Object ctx) {
+                        // ignore.
+                    }
 
-                        @Override
-                        public void deleteFailed(ManagedLedgerException ex, Object ctx) {
-                            // The method internalMarkDelete already handled the failure operation. We only need to
-                            // make sure the memory state is updated.
-                            // If the broker crashed, the non-recoverable ledger will be detected again.
-                        }
-                    }, null);
-                }
-            }
-        } finally {
-            lock.writeLock().unlock();
-        }
+                    @Override
+                    public void deleteFailed(ManagedLedgerException ex, Object ctx) {
+                        // The method internalMarkDelete already handled the failure operation. We only need to
+                        // make sure the memory state is updated.
+                        // If the broker crashed, the non-recoverable ledger will be detected again.
+                    }
+                }, null);
     }
 
     // //////////////////////////////////////////////////
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
index f028b385c516..dfd1a4679564 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedCursorTest.java
@@ -5637,6 +5637,44 @@ public class ManagedCursorTest extends MockedBookKeeperTestCase {
         ledger.close();
     }
 
+    @Test(timeOut = 20000)
+    public void testSkipNonRecoverableEntriesDoesNotHoldCursorLockWhilePersisting() throws Exception {
+        ManagedLedger ledger = factory.open("testSkipNonRecoverableEntriesDoesNotHoldCursorLockWhilePersisting",
+                new ManagedLedgerConfig());
+        ManagedCursorImpl cursor = (ManagedCursorImpl) ledger.openCursor("my-cursor");
+        Position firstPosition = ledger.addEntry("entry-0".getBytes(Encoding));
+        Position nextPosition = ledger.addEntry("entry-1".getBytes(Encoding));
+
+        AtomicBoolean readLockAvailable = new AtomicBoolean();
+        Thread skipThread = new Thread(
+                () -> cursor.skipNonRecoverableEntries(firstPosition, nextPosition),
+                "skip-non-recoverable-entries");
+        synchronized (cursor.pendingMarkDeleteOps) {
+            skipThread.start();
+            Awaitility.await().atMost(Duration.ofSeconds(5))
+                    .until(() -> skipThread.getState() == Thread.State.BLOCKED
+                            && Arrays.stream(skipThread.getStackTrace())
+                                    .anyMatch(frame -> frame.getMethodName().equals("internalAsyncMarkDelete")));
+            if (cursor.lock.readLock().tryLock()) {
+                try {
+                    readLockAvailable.set(true);
+                } finally {
+                    cursor.lock.readLock().unlock();
+                }
+            }
+        }
+
+        skipThread.join(TimeUnit.SECONDS.toMillis(5));
+        assertFalse(skipThread.isAlive(), "skip thread did not finish");
+        Awaitility.await().atMost(Duration.ofSeconds(5))
+                .until(() -> cursor.getStats().getPersistLedgerSucceed() > 0);
+        assertTrue(readLockAvailable.get(),
+                "skipNonRecoverableEntries must not hold the cursor lock while waiting to persist mark-delete");
+
+        cursor.close();
+        ledger.close();
+    }
+
     @Test
     public void testRecoverCursorWithTerminateManagedLedger() throws Exception {
         String mlName = "my_test_ledger";