You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/12 06:16:29 UTC
[PR] [fix][client] Don't apply an older segment assignment after a newer one in V5 consumer groups [pulsar]
merlimat opened a new pull request, #26858:
URL: https://github.com/apache/pulsar/pull/26858
### Motivation
**Problem.** A member of a V5 consumer group (a `StreamConsumer`, or a `CheckpointConsumer` in a consumer group)
can go back to an older segment assignment after the controller has pushed it a newer one. Its
`ScalableConsumerClient` hands each assignment the controller pushes to its listener on the I/O thread, and
`setListener` replays the current assignment on the caller's thread, so that one pushed while the consumer was
being created is not lost. Nothing serialized the two. When a push arrives just after the replay has read the
current assignment, the I/O thread hands the listener the newer assignment while the caller's thread hands it the
older one, at the same time. Both consumers converge onto the last assignment they store
(`latestAssignment = newSegments`, then a reconcile), so when the replay stores its assignment last, the consumer
goes back to the older assignment. It stays there until the controller pushes it another one: when a member joins or
leaves, the layout changes, or its controller connection reconnects.
- A `StreamConsumer` keeps subscribing to segments that moved to another member, which keeps retrying its own
subscribe to them, and does not consume the segments it was given, so nobody does.
- A grouped `CheckpointConsumer` keeps reading segments that moved to another member, which reads them too
(duplicates), and does not read the segments it was given.
The window is narrow: the replay runs once per consumer, right after its initial segments are attached, and the
push must land between the replay's read of the assignment and the consumer's store of it. Rebalances cluster
around that moment, though: members that start together, as in a deployment, push new assignments to the ones
still being created.
**Example.** Member `a` of a group gets segments 0 and 1 and is still attaching them when the group changes
twice: member `b` joins and takes segment 1, then leaves again.
1. `b` joins: the controller pushes `a` the assignment `[0]`. No listener is set yet, so it is only stored.
2. `a` has attached its segments and calls `setListener`, which replays `[0]` on the attaching thread.
3. `b` leaves: the controller pushes `a` the assignment `[0, 1]`. The I/O thread hands it to the listener at
once, alongside the replay, and `a` stores `[0, 1]`.
4. The replay stores `[0]` last: `a` releases segment 1, and nobody reads it until the next push.
`ScalableStreamConsumer.createAsyncImpl` (`ScalableCheckpointConsumer.createManagedAsync` does the same):
```java
return consumer.subscribeInitialWithRetry(initialAssignment)
.thenApply(__ -> {
consumer.startDelivery();
session.setListener(consumer); // replays the current assignment on this thread
return consumer;
})
```
`ScalableConsumerClient`:
```java
void setListener(AssignmentChangeListener listener) {
this.listener = listener;
List<ActiveSegment> current = currentAssignment.get(); // [0]
listener.onAssignmentChange(current, current); // still running when [0, 1] arrives
}
// applyAssignment, on the I/O thread
List<ActiveSegment> oldSegments = currentAssignment.getAndSet(newSegments); // [0, 1]
AssignmentChangeListener l = listener;
if (l != null) {
l.onAssignmentChange(newSegments, oldSegments); // runs alongside the replay
}
```
`ScalableStreamConsumer` (the checkpoint consumer stores it the same way):
```java
public void onAssignmentChange(List<ActiveSegment> newSegments, List<ActiveSegment> oldSegments) {
latestAssignment = newSegments; // the last store wins, and it is the replay's [0]
reconcile();
}
```
### Modifications
**Change.** `ScalableConsumerClient` now calls the listener the way `DagWatchClient` does since #26854. The calls
made from `setListener`'s thread and from the I/O thread go through a work-in-progress counter, without holding a
lock while the listener runs: the call that finds no other in progress delivers, and delivers again for each call
made meanwhile. The listener never runs twice at once and gets the newest assignment last; an update that arrives
during a delivery is delivered right after it, by the thread making that delivery, and the I/O thread does not
wait for it. Once the listener is set and nothing is in flight, each update is delivered as before, on the I/O
thread.
```java
void setListener(AssignmentChangeListener listener) {
this.listener = listener;
notifyListener();
}
private void notifyListener() {
if (pendingNotifications.getAndIncrement() != 0) {
return;
}
do {
AssignmentChangeListener l = listener;
List<ActiveSegment> segments = currentAssignment.get();
List<ActiveSegment> previous = notifiedAssignment; // starts as the initial assignment
if (l != null && segments != previous) {
notifiedAssignment = segments;
l.onAssignmentChange(segments, previous); // exceptions are caught and logged
}
} while (pendingNotifications.decrementAndGet() != 0);
}
```
`setListener` hands over the current assignment only when it is not the initial one that `start()` completed
with, which the caller applies itself; before, it always replayed it, which was redundant but harmless. An
assignment the caller already applied can still be handed over (the stream consumer retries its initial
subscribe with the session's current assignment), which stays harmless: the consumers apply assignments
idempotently. The epoch check in `onAssignmentUpdate` and the reconnect path, which feeds the subscribe
response through `onAssignmentUpdate`, are unchanged. The consumers are unchanged.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- `ScalableConsumerClientTest.testAssignmentArrivingWhileListenerRunsIsDeliveredAfterIt` sets the listener on
another thread and holds the replay inside the listener before it records the assignment, then delivers a newer
assignment through `onAssignmentUpdate`, the method the I/O thread calls. Without the fix it fails: the newer
assignment is handed over at once, alongside the replay, and the older one is recorded last. With the fix the
newer one is handed over after the replay, on the replay's thread, and is recorded last.
- `ScalableConsumerClientTest.testSetListenerDeliversAssignmentReceivedBeforeIt` checks that an assignment
received before `setListener` is still handed over.
- There is no end-to-end test: nothing in the consumers can be held between the replay's read of the assignment
and their store of it, so a broker test would hit the race only by chance.
- The `pulsar-client-v5` unit tests and all the V5 tests in `pulsar-broker` (`org.apache.pulsar.client.api.v5`,
`org.apache.pulsar.client.impl.v5`) pass.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Don't apply an older segment assignment after a newer one in V5 consumer groups [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #26858:
URL: https://github.com/apache/pulsar/pull/26858
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Don't apply an older segment assignment after a newer one in V5 consumer groups [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26858:
URL: https://github.com/apache/pulsar/pull/26858#discussion_r4250237121
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableConsumerClient.java:
##########
@@ -269,20 +280,38 @@ public void onAssignmentUpdate(ScalableConsumerAssignment assignment) {
private List<ActiveSegment> applyAssignment(ScalableConsumerAssignment assignment) {
long epoch = assignment.getLayoutEpoch();
List<ActiveSegment> newSegments = toSegmentList(assignment);
- List<ActiveSegment> oldSegments = currentAssignment.getAndSet(newSegments);
+ currentAssignment.set(newSegments);
currentEpoch = epoch;
log.info().attr("epoch", epoch).attr("segments", newSegments.size())
.log("Assignment updated");
+ notifyListener();
+ return newSegments;
+ }
- AssignmentChangeListener l = listener;
- if (l != null) {
- try {
- l.onAssignmentChange(newSegments, oldSegments);
- } catch (Exception e) {
- log.error().exception(e).log("Error in assignment change listener");
- }
+ /**
+ * Hand the listener the current assignment if it hasn't had it yet. Both the I/O thread, on every
+ * update, and {@link #setListener} call this, so the listener calls are serialized here without
+ * holding a lock while the listener runs: the call that finds no other in progress delivers, and
+ * delivers again for each call made meanwhile. The listener gets the assignments in order and the
+ * current one last, skipping any replaced before it could be delivered.
+ */
+ private void notifyListener() {
+ if (pendingNotifications.getAndIncrement() != 0) {
+ return;
}
- return newSegments;
+ do {
+ AssignmentChangeListener l = listener;
+ List<ActiveSegment> segments = currentAssignment.get();
+ List<ActiveSegment> previous = notifiedAssignment;
+ if (l != null && segments != previous) {
+ notifiedAssignment = segments;
+ try {
+ l.onAssignmentChange(segments, previous);
+ } catch (Exception e) {
+ log.error().exception(e).log("Error in assignment change listener");
Review Comment:
Fixed in 40a7a32798c: the listener calls now catch `Throwable` here, in `DagWatchClient`, and in `ScalableTopicsWatcher` (same loop), so an `Error` is logged and later updates are still delivered; each class has a new unit test where the listener throws an `Error` and the next update must still arrive (all three fail without the change).
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][client] Don't apply an older segment assignment after a newer one in V5 consumer groups [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #26858:
URL: https://github.com/apache/pulsar/pull/26858#discussion_r4246672083
##########
pulsar-client-v5/src/main/java/org/apache/pulsar/client/impl/v5/ScalableConsumerClient.java:
##########
@@ -269,20 +280,38 @@ public void onAssignmentUpdate(ScalableConsumerAssignment assignment) {
private List<ActiveSegment> applyAssignment(ScalableConsumerAssignment assignment) {
long epoch = assignment.getLayoutEpoch();
List<ActiveSegment> newSegments = toSegmentList(assignment);
- List<ActiveSegment> oldSegments = currentAssignment.getAndSet(newSegments);
+ currentAssignment.set(newSegments);
currentEpoch = epoch;
log.info().attr("epoch", epoch).attr("segments", newSegments.size())
.log("Assignment updated");
+ notifyListener();
+ return newSegments;
+ }
- AssignmentChangeListener l = listener;
- if (l != null) {
- try {
- l.onAssignmentChange(newSegments, oldSegments);
- } catch (Exception e) {
- log.error().exception(e).log("Error in assignment change listener");
- }
+ /**
+ * Hand the listener the current assignment if it hasn't had it yet. Both the I/O thread, on every
+ * update, and {@link #setListener} call this, so the listener calls are serialized here without
+ * holding a lock while the listener runs: the call that finds no other in progress delivers, and
+ * delivers again for each call made meanwhile. The listener gets the assignments in order and the
+ * current one last, skipping any replaced before it could be delivered.
+ */
+ private void notifyListener() {
+ if (pendingNotifications.getAndIncrement() != 0) {
+ return;
}
- return newSegments;
+ do {
+ AssignmentChangeListener l = listener;
+ List<ActiveSegment> segments = currentAssignment.get();
+ List<ActiveSegment> previous = notifiedAssignment;
+ if (l != null && segments != previous) {
+ notifiedAssignment = segments;
+ try {
+ l.onAssignmentChange(segments, previous);
+ } catch (Exception e) {
+ log.error().exception(e).log("Error in assignment change listener");
Review Comment:
**[NIT] an Error from the listener leaves the counter non-zero and stops all later deliveries**
Only `Exception` is caught, so if the listener throws an `Error` (e.g. `AssertionError`), the `decrementAndGet()` at the end of the loop is skipped. `pendingNotifications` then stays above zero, every later `notifyListener()` returns at the first check, and the consumer never sees another assignment, including after a reconnect. Previously a stray `Error` only failed that one callback.
It is very unlikely, and `DagWatchClient` has the same shape, so feel free to leave it. If you want to harden it, wrap the delivery in `try { ... } finally { }` or catch `Throwable` here (and in `DagWatchClient`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org