You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/12 19:28:55 UTC
[PR] [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers [pulsar]
merlimat opened a new pull request, #26874:
URL: https://github.com/apache/pulsar/pull/26874
### Motivation
**Problem.** `ScalableStreamConsumer` and `ScalableQueueConsumer` move their per-segment v4 consumers onto the latest controller assignment (stream) or DAG layout (queue). They use the same single-flight reconcile loop as `ScalableCheckpointConsumer`, where @lhotari's review of #26837 found two problems. Both are present here:
1. **Several retry chains.** The guard is cleared before the retry is scheduled. An update that arrives during the backoff starts its own reconcile, and if that fails too, it schedules a second retry while the first is still pending. Repeated failures therefore leave several independent retry chains, for example when a stream consumer's subscribe keeps being rejected with `ConsumerBusy` because another member still holds the segment. `Backoff` keeps its state in plain fields, yet every chain calls `next()`/`reset()` from whichever thread completes a subscribe or runs the retry.
2. **Stall behind a pending future.** The guard stays set until every subscribe or drain of the reconcile completes, so newer updates aren't applied in the meantime.
- Queue consumer: the segments a split creates aren't subscribed, and so aren't consumed, while another segment's subscribe keeps reconnecting. On every attempt, that can last up to the operation timeout (30 s by default).
- Stream consumer: a newly assigned segment isn't subscribed until the application acks what it holds from a revoked segment, because `drainAndClose` is deliberately unbounded.
**Example.** The current code path, from the stream consumer. The queue consumer's loop is the same, with `latestLayout` and `subscribeSegments`:
```java
private void reconcile() {
if (closed || !reconcileInProgress.compareAndSet(false, true)) {
return;
}
List<ActiveSegment> target = latestAssignment;
// (2) the guard is held until every subscribe/drain future completes
subscribeAssigned(target).whenComplete((__, ex) -> {
reconcileInProgress.set(false); // (1) cleared before the retry is scheduled
...
evictFailedSegmentConsumers();
Duration delay = reconcileBackoff.next(); // (1) unsynchronized, from any thread
scheduler().schedule(this::reconcile, delay.toMillis(), TimeUnit.MILLISECONDS);
});
}
```
Segment 1's subscribe is rejected, so retry R1 is scheduled. The same assignment is pushed twice while R1 waits: each push reconciles at once, fails, and schedules another retry, leaving three retry chains. On master, `keepsAtMostOneRetryScheduled` fails with `expected [1] but found [3]` for both consumers.
### Modifications
**Change.**
- **Both consumers schedule at most one retry.** A `retryScheduled` flag and the `Backoff` are both guarded by the consumer's monitor. An update that arrives during the backoff still reconciles right away; if that fails too, it doesn't schedule a second retry:
```java
private void scheduleRetry(Throwable ex) {
Duration delay;
synchronized (this) {
if (retryScheduled) {
return;
}
retryScheduled = true;
delay = reconcileBackoff.next();
}
log.warn().attr("delayMs", delay.toMillis()).exceptionMessage(ex)
.log("Failed to apply layout update, retrying after backoff");
scheduler().schedule(() -> {
synchronized (this) {
retryScheduled = false;
}
reconcile();
}, delay.toMillis(), TimeUnit.MILLISECONDS);
}
```
- **The queue consumer applies a layout as soon as it arrives**, as the checkpoint consumer does since #26837. With a `reconcileRequested` flag and a loop, one thread applies the latest layout at a time, and applies it again if asked to meanwhile. Failed subscribes are evicted at the start of each apply. The guard now covers only applying a layout, not waiting for the subscribes:
```java
private void reconcile() {
reconcileRequested.set(true);
// One thread applies at a time, and applies again if asked to meanwhile.
while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
reconcileRequested.set(false);
evictFailedSegmentConsumers();
subscribeSegments(latestLayout).whenComplete((__, ex) -> {
if (ex == null) {
synchronized (this) {
reconcileBackoff.reset();
}
} else if (!closed) {
scheduleRetry(ex);
}
});
reconcileInProgress.set(false);
}
}
```
This is safe for the queue consumer because it has no release drain, and its only per-segment state is the map of subscribe futures. A newer layout either keeps an in-flight subscribe (`computeIfAbsent`) or chains the close of a segment that left the DAG onto it.
- **The stream consumer still waits** for a reconcile's releases and subscribes before applying a newer assignment, and the reason is now documented on `reconcile()`. Its release state is keyed by segment id: `drainingConsumers`, `pausedSegments`, `pendingDrainAcks`, `segmentReceiveEpoch`, `sharedSegmentUnacked`, `lastCumulativeAcked` and `latestDelivered`. The PIP-486 bucket handoff relies on a segment's drain finishing before the segment is subscribed again. Applying a newer assignment while a drain or subscribe is in flight would:
- **Re-subscribe a draining segment.** A segment that moved away and back would be subscribed again while its previous consumer still drains. When the drain completes, its cleanup removes the segment's receive epoch from under the new consumer, whose receive loop then discards every message and stops re-arming. Swapping in the checkpoint-style loop reproduces this: the segment subscribed again never delivers.
- **Start a second drain of the same segment** while its mode-flip re-subscribe still waits for the first one. This overwrites `drainingConsumers`, and the first drain's completion then un-pauses the segment and clears `pendingDrainAcks` under the second.
- **Drain a segment whose subscribe is still pending.** Its receive loop then arms with the post-drain epoch, and can hand the application a message just before the consumer is closed.
Removing the stall there would need per-segment serialization, which is out of scope for this fix. The loop still retries `ConsumerBusy`/`ConsumerAssignError` rejections, now with at most one retry pending.
### Verifying this change
This change added tests and can be verified as follows. The new unit tests are written in the style of `ScalableCheckpointConsumerReconcileTest`, with mocked `PulsarClientImpl`, scheduler, `ScalableConsumerClient` and `DagWatchClient`:
- `ScalableStreamConsumerReconcileTest.keepsAtMostOneRetryScheduled` and `ScalableQueueConsumerReconcileTest.keepsAtMostOneRetryScheduled` fail on master, where 3 retries are scheduled instead of 1.
- `ScalableQueueConsumerReconcileTest.subscribesNewSegmentsWhileAnotherSubscribeIsPending` fails on master: the split's segments aren't subscribed while segment 1's subscribe is pending.
- `ScalableStreamConsumerReconcileTest.resubscribesASegmentOnlyAfterItsReleaseDrained` passes both before and after this change, and fails with the checkpoint-style loop. It guards the drain-before-resubscribe ordering that the stream consumer keeps.
`./gradlew quickCheck` passes. With `-PtestRetryCount=0 -PtestFailFast=false`, these also pass:
- all 174 `pulsar-client-v5` tests;
- 51 tests in these `pulsar-broker` V5 suites: `V5StreamConsumer{Basic,AutoReconnect,DagReplay,AckWatermark}`, `V5QueueConsumer{Basic,MultiSegmentAck}`, `V5MultiTopic{Stream,Queue}Consumer`, `V5MultipleConsumers`, `V5EntryBucketDispatch`, `V5SegmentRebucket`, `V5AutoRebucket`, `V5SegmentSplit`, `V5SegmentMerge`, `V5DAGFollowing`, `V5DagWatchAutoReconnect` and `V5NamespaceConsumerTopicDeletion`.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26874:
URL: https://github.com/apache/pulsar/pull/26874#discussion_r4252234873
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumer.java:
##########
@@ -599,13 +614,30 @@ private void reconcile() {
}
// Evict failed subscribe futures so the next attempt can re-try them.
evictFailedSegmentConsumers();
- Duration delay = reconcileBackoff.next();
- log.warn().attr("delayMs", delay.toMillis()).exceptionMessage(ex)
- .log("Failed to apply assignment update, retrying after backoff");
- scheduler().schedule(this::reconcile, delay.toMillis(), TimeUnit.MILLISECONDS);
+ scheduleRetry(ex);
});
}
+ /** Retry the reconcile after a backoff, with at most one retry scheduled. */
+ private void scheduleRetry(Throwable ex) {
+ Duration delay;
+ synchronized (this) {
+ if (retryScheduled) {
Review Comment:
Done in 6b41fb577ef, as suggested: a success marks the backoff reset (`backoffResetSinceRetry = retryScheduled`), a later failure then replaces the pending retry, and a generation token makes the replaced task do nothing; covered by `retriesANewFailureWithoutWaitingOutAnEarlierBackoff` (the checkpoint consumer is not changed in this PR).
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
}
+ /**
+ * Converge the per-segment consumers onto {@link #latestLayout}. A layout is applied as soon as
+ * it arrives, not after the consumers of the previous one have subscribed: a subscribe that keeps
+ * reconnecting must not delay subscribing the segments a split creates. A consumer that fails to
+ * subscribe is evicted and retried with backoff.
+ */
private void reconcile() {
- if (closed) {
- return;
- }
- if (!reconcileInProgress.compareAndSet(false, true)) {
- // A reconcile is already in progress; it will observe the updated latestLayout
- // when it finishes and re-run if needed.
- return;
- }
- ClientSegmentLayout target = latestLayout;
- subscribeSegments(target).whenComplete((__, ex) -> {
+ reconcileRequested.set(true);
+ // One thread applies at a time, and applies again if asked to meanwhile.
+ while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
+ reconcileRequested.set(false);
+ evictFailedSegmentConsumers();
+ subscribeSegments(latestLayout).whenComplete((__, ex) -> {
+ if (ex == null) {
+ synchronized (this) {
+ reconcileBackoff.reset();
+ }
+ } else if (!closed) {
+ scheduleRetry(ex);
+ }
+ });
reconcileInProgress.set(false);
- if (closed) {
+ }
+ }
+
+ /** Retry the segments that failed to subscribe after a backoff, with at most one retry scheduled. */
+ private void scheduleRetry(Throwable ex) {
+ Duration delay;
+ synchronized (this) {
+ if (retryScheduled) {
Review Comment:
Done in 6b41fb577ef with the same change; a success only marks the backoff as reset and never cancels the pending retry, and `ScalableQueueConsumerReconcileTest.retriesANewFailureWithoutWaitingOutAnEarlierBackoff` covers it.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumer.java:
##########
@@ -599,13 +614,30 @@ private void reconcile() {
}
// Evict failed subscribe futures so the next attempt can re-try them.
evictFailedSegmentConsumers();
- Duration delay = reconcileBackoff.next();
- log.warn().attr("delayMs", delay.toMillis()).exceptionMessage(ex)
- .log("Failed to apply assignment update, retrying after backoff");
- scheduler().schedule(this::reconcile, delay.toMillis(), TimeUnit.MILLISECONDS);
+ scheduleRetry(ex);
Review Comment:
Done in 6b41fb577ef: a failed reconcile overtaken by a newer assignment now re-runs right away (as on success), and failed subscribes are evicted at the start of `reconcile()`, inside the guard; covered by `appliesAnAssignmentThatArrivedDuringAFailedReconcileRightAway`.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
}
+ /**
+ * Converge the per-segment consumers onto {@link #latestLayout}. A layout is applied as soon as
+ * it arrives, not after the consumers of the previous one have subscribed: a subscribe that keeps
+ * reconnecting must not delay subscribing the segments a split creates. A consumer that fails to
+ * subscribe is evicted and retried with backoff.
+ */
private void reconcile() {
- if (closed) {
- return;
- }
- if (!reconcileInProgress.compareAndSet(false, true)) {
- // A reconcile is already in progress; it will observe the updated latestLayout
- // when it finishes and re-run if needed.
- return;
- }
- ClientSegmentLayout target = latestLayout;
- subscribeSegments(target).whenComplete((__, ex) -> {
+ reconcileRequested.set(true);
+ // One thread applies at a time, and applies again if asked to meanwhile.
+ while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
+ reconcileRequested.set(false);
+ evictFailedSegmentConsumers();
+ subscribeSegments(latestLayout).whenComplete((__, ex) -> {
+ if (ex == null) {
+ synchronized (this) {
+ reconcileBackoff.reset();
+ }
+ } else if (!closed) {
+ scheduleRetry(ex);
+ }
+ });
reconcileInProgress.set(false);
Review Comment:
Done in 6b41fb577ef: the apply now releases the guard in a `finally` block.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
Review Comment:
Done in 6b41fb577ef: trimmed to the first sentence, as in the checkpoint consumer.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26874:
URL: https://github.com/apache/pulsar/pull/26874#discussion_r4252126470
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumer.java:
##########
@@ -599,13 +614,30 @@ private void reconcile() {
}
// Evict failed subscribe futures so the next attempt can re-try them.
evictFailedSegmentConsumers();
- Duration delay = reconcileBackoff.next();
- log.warn().attr("delayMs", delay.toMillis()).exceptionMessage(ex)
- .log("Failed to apply assignment update, retrying after backoff");
- scheduler().schedule(this::reconcile, delay.toMillis(), TimeUnit.MILLISECONDS);
+ scheduleRetry(ex);
Review Comment:
On success this re-runs right away when `latestAssignment != target`, but on failure it always waits for the backoff. An assignment that arrived during this reconcile is then applied only when the retry fires, up to 30 s later. That holds even if it releases a segment another member is waiting on, or no longer includes the segment that failed. This is pre-existing, but it's the same kind of stall this PR removes elsewhere. The reconcile has already completed, so re-running now is as safe as on success:
```java
evictFailedSegmentConsumers();
if (latestAssignment != target) {
reconcile(); // its own failure path schedules the retry if this fails too
} else {
scheduleRetry(ex);
}
```
Minor, while you're here: the eviction runs after `reconcileInProgress.set(false)` (L601). A reconcile started on another thread in between reuses the stale failed future. Evicting before releasing the guard, or at the start of `reconcile()` as the queue consumer now does, avoids that wasted round.
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumer.java:
##########
@@ -599,13 +614,30 @@ private void reconcile() {
}
// Evict failed subscribe futures so the next attempt can re-try them.
evictFailedSegmentConsumers();
- Duration delay = reconcileBackoff.next();
- log.warn().attr("delayMs", delay.toMillis()).exceptionMessage(ex)
- .log("Failed to apply assignment update, retrying after backoff");
- scheduler().schedule(this::reconcile, delay.toMillis(), TimeUnit.MILLISECONDS);
+ scheduleRetry(ex);
});
}
+ /** Retry the reconcile after a backoff, with at most one retry scheduled. */
+ private void scheduleRetry(Throwable ex) {
+ Duration delay;
+ synchronized (this) {
+ if (retryScheduled) {
Review Comment:
`retryScheduled` stays set until the pending retry fires, even after a successful reconcile has `reset()` the backoff. Suppose failures grow the backoff to N s, then a reconcile triggered by an update succeeds. A new failure after that (e.g. a newly assigned segment whose previous owner is still releasing it) returns here and waits for the stale N-second retry, instead of ~100 ms as on master. I reproduced this with a throwaway test: after 6 failed retries, the new failure was folded into a retry pending for ~6.5 s. It's bounded, but it slows rebalances while membership is changing.
One option is to let a failure replace the pending retry when the backoff was reset after it was scheduled, with a generation token so the replaced task does nothing:
```java
// success path, under the lock: reconcileBackoff.reset(); backoffResetSinceRetry = retryScheduled;
synchronized (this) {
if (retryScheduled && !backoffResetSinceRetry) {
return;
}
retryScheduled = true;
backoffResetSinceRetry = false;
generation = ++retryGeneration;
delay = reconcileBackoff.next();
}
// ...
scheduler().schedule(() -> {
synchronized (this) {
if (generation != retryGeneration) {
return; // replaced by a sooner retry
}
retryScheduled = false;
}
reconcile();
}, delay.toMillis(), TimeUnit.MILLISECONDS);
```
The checkpoint consumer has the same pattern, so a follow-up covering all three consumers would also work.
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumerReconcileTest.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.api.proto.ScalableTopicDAG;
+import org.apache.pulsar.common.api.proto.SegmentState;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableQueueConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled retries, run by the test. */
+ private final ArrayDeque<Runnable> retries = new ArrayDeque<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ retries.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ retries.add(invocation.getArgument(0));
+ return null;
+ });
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps failing with a transient error.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new PulsarClientException("transient"))
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(retries.size(), 1, "the failed subscribe must be retried");
+
+ // The layout is pushed again while the retry waits: each push retries segment 1 right
+ // away, which fails again, without scheduling a second retry.
+ listener[0].onLayoutChange(split, split);
+ listener[0].onLayoutChange(split, split);
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(retries.size(), 1, "at most one retry must be scheduled");
+
+ retries.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(retries.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
Review Comment:
Could also assert that the extra applies and retries don't recreate the segments that already subscribed (`attempts.get(0L)` and `attempts.get(2L)` stay 1). The checkpoint test does this with "attached readers are not recreated".
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumerReconcileTest.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.PulsarClientException.ConsumerBusyException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ClientConfigurationData;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.impl.v5.SegmentRouter.ActiveSegment;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.scalable.HashRange;
+import org.apache.pulsar.common.scalable.SegmentTopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableStreamConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled tasks (reconcile retries and release drain polls), run by the test. */
+ private final ArrayDeque<Runnable> scheduled = new ArrayDeque<>();
+ /** Messages handed to the application. */
+ private final List<MessageV5<byte[]>> delivered = new ArrayList<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+ private ScalableConsumerClient session;
+ private ScalableConsumerClient.AssignmentChangeListener listener;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ scheduled.clear();
+ delivered.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getConfiguration()).thenReturn(new ClientConfigurationData());
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ scheduled.add(invocation.getArgument(0));
+ return null;
+ });
+ when(scheduler.schedule(any(Callable.class), anyLong(), any())).thenAnswer(invocation -> {
+ Callable<?> task = invocation.getArgument(0);
+ scheduled.add(() -> {
+ try {
+ task.call();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ });
+ return null;
+ });
+ session = mock(ScalableConsumerClient.class);
+ doAnswer(invocation -> {
+ listener = invocation.getArgument(0);
+ return null;
+ }).when(session).setListener(any());
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps being rejected, as while its previous owner still holds it.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new ConsumerBusyException("busy"))
+ : CompletableFuture.completedFuture(consumer(new CompletableFuture<>())));
+ ScalableStreamConsumer<byte[]> consumer = create(List.of(segment(0)));
+ try {
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0)));
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(scheduled.size(), 1, "the rejected subscribe must be retried");
+
+ // The assignment is pushed again while the retry waits: each push retries segment 1
+ // right away, which fails again, without scheduling a second retry.
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0), segment(1)));
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0), segment(1)));
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(scheduled.size(), 1, "at most one retry must be scheduled");
+
+ scheduled.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(scheduled.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
+ } finally {
+ consumer.close();
+ }
+ }
+
+ @Test
+ public void resubscribesASegmentOnlyAfterItsReleaseDrained() throws Exception {
+ // Segment 0's first consumer hands the application a message; its next one, another.
+ CompletableFuture<Message<byte[]>> firstReceive = new CompletableFuture<>();
+ CompletableFuture<Message<byte[]>> secondReceive = new CompletableFuture<>();
+ subscribeSegments((segmentId, attempt) -> CompletableFuture.completedFuture(consumer(
+ segmentId != 0 ? new CompletableFuture<>() : attempt == 1 ? firstReceive : secondReceive)));
+ ScalableStreamConsumer<byte[]> consumer = create(List.of(segment(0)));
+ try {
+ firstReceive.complete(message());
+ assertEquals(delivered.size(), 1);
+
+ // Segment 0 moves to another member and back while the application holds its message:
+ // the assignment that gives it back is applied once the release has drained.
+ listener.onAssignmentChange(List.of(segment(1)), List.of(segment(0)));
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(1)));
Review Comment:
The main risk of the flag design is a lost retry. A retry that fires while a reconcile holds the guard (as here, during the drain) does nothing, and only the in-flight reconcile's failure schedules a new one. That works: in a throwaway test the retry fired mid-drain, and a ~200 ms retry was scheduled once the drain completed. It's still worth a test. For example: reject segment 1, move segment 0 away while the app holds its message, run the pending retry, ack, run the drain polls, then assert a retry is scheduled.
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
}
+ /**
+ * Converge the per-segment consumers onto {@link #latestLayout}. A layout is applied as soon as
+ * it arrives, not after the consumers of the previous one have subscribed: a subscribe that keeps
+ * reconnecting must not delay subscribing the segments a split creates. A consumer that fails to
+ * subscribe is evicted and retried with backoff.
+ */
private void reconcile() {
- if (closed) {
- return;
- }
- if (!reconcileInProgress.compareAndSet(false, true)) {
- // A reconcile is already in progress; it will observe the updated latestLayout
- // when it finishes and re-run if needed.
- return;
- }
- ClientSegmentLayout target = latestLayout;
- subscribeSegments(target).whenComplete((__, ex) -> {
+ reconcileRequested.set(true);
+ // One thread applies at a time, and applies again if asked to meanwhile.
+ while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
+ reconcileRequested.set(false);
+ evictFailedSegmentConsumers();
+ subscribeSegments(latestLayout).whenComplete((__, ex) -> {
Review Comment:
Follow-up idea, not blocking: failures are only seen through the `allOf` in `subscribeSegments`. So a segment whose subscribe fails fast is retried only after every other subscribe in the same apply has settled. If a split's segment 1 fails while segment 2 keeps reconnecting, segment 1 can wait up to the operation timeout, unless another layout arrives (confirmed with a throwaway test). That's the stall this PR fixes, moved to the retry path. One way to close it: call `scheduleRetry` from each newly created segment future (e.g. in the `computeIfAbsent` mapping function), and keep `allOf` only to reset the backoff.
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
Review Comment:
Nit: the comment above (L315-318) is now stale: an in-progress apply no longer waits for its subscribes before re-reading `latestLayout`, and failures share a single pending retry. The checkpoint consumer trimmed its version to the first sentence.
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
}
+ /**
+ * Converge the per-segment consumers onto {@link #latestLayout}. A layout is applied as soon as
+ * it arrives, not after the consumers of the previous one have subscribed: a subscribe that keeps
+ * reconnecting must not delay subscribing the segments a split creates. A consumer that fails to
+ * subscribe is evicted and retried with backoff.
+ */
private void reconcile() {
- if (closed) {
- return;
- }
- if (!reconcileInProgress.compareAndSet(false, true)) {
- // A reconcile is already in progress; it will observe the updated latestLayout
- // when it finishes and re-run if needed.
- return;
- }
- ClientSegmentLayout target = latestLayout;
- subscribeSegments(target).whenComplete((__, ex) -> {
+ reconcileRequested.set(true);
+ // One thread applies at a time, and applies again if asked to meanwhile.
+ while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
+ reconcileRequested.set(false);
+ evictFailedSegmentConsumers();
+ subscribeSegments(latestLayout).whenComplete((__, ex) -> {
+ if (ex == null) {
+ synchronized (this) {
+ reconcileBackoff.reset();
+ }
+ } else if (!closed) {
+ scheduleRetry(ex);
+ }
+ });
reconcileInProgress.set(false);
- if (closed) {
+ }
+ }
+
+ /** Retry the segments that failed to subscribe after a backoff, with at most one retry scheduled. */
+ private void scheduleRetry(Throwable ex) {
+ Duration delay;
+ synchronized (this) {
+ if (retryScheduled) {
Review Comment:
Same issue as in `ScalableStreamConsumer#scheduleRetry` (see my comment there), reproduced here too. Right after a successful apply reset the backoff, a new failure of a split's segment was folded into a retry pending for ~6.2 s. Note that a success must not just cancel the pending retry here. Applies overlap, so an older apply's success can complete after a newer apply's failure and would drop the only retry.
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumer.java:
##########
@@ -315,36 +320,49 @@ public void onLayoutChange(ClientSegmentLayout newLayout, ClientSegmentLayout ol
reconcile();
}
+ /**
+ * Converge the per-segment consumers onto {@link #latestLayout}. A layout is applied as soon as
+ * it arrives, not after the consumers of the previous one have subscribed: a subscribe that keeps
+ * reconnecting must not delay subscribing the segments a split creates. A consumer that fails to
+ * subscribe is evicted and retried with backoff.
+ */
private void reconcile() {
- if (closed) {
- return;
- }
- if (!reconcileInProgress.compareAndSet(false, true)) {
- // A reconcile is already in progress; it will observe the updated latestLayout
- // when it finishes and re-run if needed.
- return;
- }
- ClientSegmentLayout target = latestLayout;
- subscribeSegments(target).whenComplete((__, ex) -> {
+ reconcileRequested.set(true);
+ // One thread applies at a time, and applies again if asked to meanwhile.
+ while (!closed && reconcileRequested.get() && reconcileInProgress.compareAndSet(false, true)) {
+ reconcileRequested.set(false);
+ evictFailedSegmentConsumers();
+ subscribeSegments(latestLayout).whenComplete((__, ex) -> {
+ if (ex == null) {
+ synchronized (this) {
+ reconcileBackoff.reset();
+ }
+ } else if (!closed) {
+ scheduleRetry(ex);
+ }
+ });
reconcileInProgress.set(false);
Review Comment:
Nit: consider `try { ... } finally { reconcileInProgress.set(false); }`. If `subscribeSegments` ever throws synchronously, the guard stays set and every later layout is silently ignored, because `DagWatchClient.notifyListener` only logs. This is pre-existing, but now that the guard covers only synchronous work, try/finally is the natural shape.
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumerReconcileTest.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.api.proto.ScalableTopicDAG;
+import org.apache.pulsar.common.api.proto.SegmentState;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableQueueConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled retries, run by the test. */
+ private final ArrayDeque<Runnable> retries = new ArrayDeque<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ retries.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ retries.add(invocation.getArgument(0));
+ return null;
+ });
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps failing with a transient error.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new PulsarClientException("transient"))
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(retries.size(), 1, "the failed subscribe must be retried");
+
+ // The layout is pushed again while the retry waits: each push retries segment 1 right
+ // away, which fails again, without scheduling a second retry.
+ listener[0].onLayoutChange(split, split);
+ listener[0].onLayoutChange(split, split);
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(retries.size(), 1, "at most one retry must be scheduled");
+
+ retries.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(retries.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
+ } finally {
+ consumer.close();
+ }
+ }
+
+ @Test
+ public void subscribesNewSegmentsWhileAnotherSubscribeIsPending() throws Exception {
+ // Segment 1's subscribe doesn't complete, as while it keeps reconnecting.
+ CompletableFuture<Consumer<byte[]>> segment1Subscribe = new CompletableFuture<>();
+ subscribeSegments((segmentId, attempt) -> segmentId == 1 ? segment1Subscribe
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ // Segment 0 splits into 1 and 2, then segment 2 splits into 3 and 4 while segment 1's
+ // subscribe is still pending.
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ listener[0].onLayoutChange(layout(2), split);
+ assertEquals(attempts.get(3L), 1, "a new segment must be subscribed right away");
+ assertEquals(attempts.get(4L), 1, "a new segment must be subscribed right away");
Review Comment:
Suggest also asserting `assertEquals(attempts.get(1L), 1, "an in-flight subscribe must be kept, not duplicated")`. Reusing the pending subscribe via `computeIfAbsent` is the core safety argument for applying layouts without waiting. The other half deserves a test too: a segment that leaves the DAG while its subscribe is pending gets closed once the subscribe completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26874:
URL: https://github.com/apache/pulsar/pull/26874#discussion_r4252236315
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumerReconcileTest.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.api.proto.ScalableTopicDAG;
+import org.apache.pulsar.common.api.proto.SegmentState;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableQueueConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled retries, run by the test. */
+ private final ArrayDeque<Runnable> retries = new ArrayDeque<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ retries.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ retries.add(invocation.getArgument(0));
+ return null;
+ });
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps failing with a transient error.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new PulsarClientException("transient"))
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(retries.size(), 1, "the failed subscribe must be retried");
+
+ // The layout is pushed again while the retry waits: each push retries segment 1 right
+ // away, which fails again, without scheduling a second retry.
+ listener[0].onLayoutChange(split, split);
+ listener[0].onLayoutChange(split, split);
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(retries.size(), 1, "at most one retry must be scheduled");
+
+ retries.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(retries.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
+ } finally {
+ consumer.close();
+ }
+ }
+
+ @Test
+ public void subscribesNewSegmentsWhileAnotherSubscribeIsPending() throws Exception {
+ // Segment 1's subscribe doesn't complete, as while it keeps reconnecting.
+ CompletableFuture<Consumer<byte[]>> segment1Subscribe = new CompletableFuture<>();
+ subscribeSegments((segmentId, attempt) -> segmentId == 1 ? segment1Subscribe
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ // Segment 0 splits into 1 and 2, then segment 2 splits into 3 and 4 while segment 1's
+ // subscribe is still pending.
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ listener[0].onLayoutChange(layout(2), split);
+ assertEquals(attempts.get(3L), 1, "a new segment must be subscribed right away");
+ assertEquals(attempts.get(4L), 1, "a new segment must be subscribed right away");
Review Comment:
Done in 6b41fb577ef: added the assertion, plus `closesASegmentThatLeftTheLayoutOnceItsSubscribeCompletes` for a segment that leaves the layout while its subscribe is pending.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableQueueConsumerReconcileTest.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.PulsarClientException;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.api.proto.ScalableTopicDAG;
+import org.apache.pulsar.common.api.proto.SegmentState;
+import org.apache.pulsar.common.naming.TopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableQueueConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled retries, run by the test. */
+ private final ArrayDeque<Runnable> retries = new ArrayDeque<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ retries.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ retries.add(invocation.getArgument(0));
+ return null;
+ });
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps failing with a transient error.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new PulsarClientException("transient"))
+ : CompletableFuture.completedFuture(consumer()));
+ DagWatchClient.LayoutChangeListener[] listener = new DagWatchClient.LayoutChangeListener[1];
+ ClientSegmentLayout initial = layout(0);
+ ScalableQueueConsumer<byte[]> consumer = create(initial, listener);
+ try {
+ ClientSegmentLayout split = layout(1);
+ listener[0].onLayoutChange(split, initial);
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(retries.size(), 1, "the failed subscribe must be retried");
+
+ // The layout is pushed again while the retry waits: each push retries segment 1 right
+ // away, which fails again, without scheduling a second retry.
+ listener[0].onLayoutChange(split, split);
+ listener[0].onLayoutChange(split, split);
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(retries.size(), 1, "at most one retry must be scheduled");
+
+ retries.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(retries.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
Review Comment:
Done in 6b41fb577ef: `keepsAtMostOneRetryScheduled` now asserts that segments 0 and 2 are subscribed only once.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-client-v5/src/test/java/org/apache/pulsar/client/impl/v5/ScalableStreamConsumerReconcileTest.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.impl.v5;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import com.google.common.util.concurrent.MoreExecutors;
+import io.netty.util.Timer;
+import java.lang.reflect.Proxy;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.function.BiFunction;
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.MessageId;
+import org.apache.pulsar.client.api.PulsarClientException.AlreadyClosedException;
+import org.apache.pulsar.client.api.PulsarClientException.ConsumerBusyException;
+import org.apache.pulsar.client.api.v5.schema.Schema;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.client.impl.conf.ClientConfigurationData;
+import org.apache.pulsar.client.impl.conf.ConsumerConfigurationData;
+import org.apache.pulsar.client.impl.v5.SegmentRouter.ActiveSegment;
+import org.apache.pulsar.client.util.ExecutorProvider;
+import org.apache.pulsar.client.util.ScheduledExecutorProvider;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.scalable.HashRange;
+import org.apache.pulsar.common.scalable.SegmentTopicName;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class ScalableStreamConsumerReconcileTest {
+
+ private static final TopicName TOPIC = TopicName.get("topic://public/default/reconcile-test");
+
+ /** Segment subscribe attempts per segment. */
+ private final Map<Long, Integer> attempts = new HashMap<>();
+ /** Scheduled tasks (reconcile retries and release drain polls), run by the test. */
+ private final ArrayDeque<Runnable> scheduled = new ArrayDeque<>();
+ /** Messages handed to the application. */
+ private final List<MessageV5<byte[]>> delivered = new ArrayList<>();
+ private ExecutorService externalExecutor;
+ private PulsarClientImpl v4;
+ private PulsarClientV5 client;
+ private ScalableConsumerClient session;
+ private ScalableConsumerClient.AssignmentChangeListener listener;
+
+ @BeforeMethod
+ public void setup() {
+ attempts.clear();
+ scheduled.clear();
+ delivered.clear();
+ externalExecutor = MoreExecutors.newDirectExecutorService();
+ v4 = mock(PulsarClientImpl.class);
+ client = mock(PulsarClientV5.class);
+ ExecutorProvider external = mock(ExecutorProvider.class);
+ ScheduledExecutorProvider scheduledProvider = mock(ScheduledExecutorProvider.class);
+ ScheduledExecutorService scheduler = mock(ScheduledExecutorService.class);
+ when(client.v4Client()).thenReturn(v4);
+ when(v4.externalExecutorProvider()).thenReturn(external);
+ when(external.getExecutor()).thenReturn(externalExecutor);
+ when(v4.timer()).thenReturn(mock(Timer.class));
+ when(v4.getConfiguration()).thenReturn(new ClientConfigurationData());
+ when(v4.getScheduledExecutorProvider()).thenReturn(scheduledProvider);
+ when(scheduledProvider.getExecutor()).thenReturn(scheduler);
+ when(scheduler.schedule(any(Runnable.class), anyLong(), any())).thenAnswer(invocation -> {
+ scheduled.add(invocation.getArgument(0));
+ return null;
+ });
+ when(scheduler.schedule(any(Callable.class), anyLong(), any())).thenAnswer(invocation -> {
+ Callable<?> task = invocation.getArgument(0);
+ scheduled.add(() -> {
+ try {
+ task.call();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ });
+ return null;
+ });
+ session = mock(ScalableConsumerClient.class);
+ doAnswer(invocation -> {
+ listener = invocation.getArgument(0);
+ return null;
+ }).when(session).setListener(any());
+ }
+
+ @AfterMethod(alwaysRun = true)
+ public void cleanup() {
+ externalExecutor.shutdown();
+ }
+
+ @Test
+ public void keepsAtMostOneRetryScheduled() throws Exception {
+ // Segment 1's subscribe keeps being rejected, as while its previous owner still holds it.
+ subscribeSegments((segmentId, attempt) -> segmentId == 1
+ ? CompletableFuture.failedFuture(new ConsumerBusyException("busy"))
+ : CompletableFuture.completedFuture(consumer(new CompletableFuture<>())));
+ ScalableStreamConsumer<byte[]> consumer = create(List.of(segment(0)));
+ try {
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0)));
+ assertEquals(attempts.get(1L), 1);
+ assertEquals(scheduled.size(), 1, "the rejected subscribe must be retried");
+
+ // The assignment is pushed again while the retry waits: each push retries segment 1
+ // right away, which fails again, without scheduling a second retry.
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0), segment(1)));
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(0), segment(1)));
+ assertEquals(attempts.get(1L), 3);
+ assertEquals(scheduled.size(), 1, "at most one retry must be scheduled");
+
+ scheduled.poll().run();
+ assertEquals(attempts.get(1L), 4);
+ assertEquals(scheduled.size(), 1, "the retry must be rescheduled while the subscribe keeps failing");
+ } finally {
+ consumer.close();
+ }
+ }
+
+ @Test
+ public void resubscribesASegmentOnlyAfterItsReleaseDrained() throws Exception {
+ // Segment 0's first consumer hands the application a message; its next one, another.
+ CompletableFuture<Message<byte[]>> firstReceive = new CompletableFuture<>();
+ CompletableFuture<Message<byte[]>> secondReceive = new CompletableFuture<>();
+ subscribeSegments((segmentId, attempt) -> CompletableFuture.completedFuture(consumer(
+ segmentId != 0 ? new CompletableFuture<>() : attempt == 1 ? firstReceive : secondReceive)));
+ ScalableStreamConsumer<byte[]> consumer = create(List.of(segment(0)));
+ try {
+ firstReceive.complete(message());
+ assertEquals(delivered.size(), 1);
+
+ // Segment 0 moves to another member and back while the application holds its message:
+ // the assignment that gives it back is applied once the release has drained.
+ listener.onAssignmentChange(List.of(segment(1)), List.of(segment(0)));
+ listener.onAssignmentChange(List.of(segment(0), segment(1)), List.of(segment(1)));
Review Comment:
Added `keepsTheRetryThatFiresWhileASegmentDrains` in 6b41fb577ef: the retry fires mid-drain and does nothing, and once the release drained the reconcile's own failure schedules the next retry.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #26874:
URL: https://github.com/apache/pulsar/pull/26874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org