You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/07 22:36:56 UTC
[I] [Bug] V5 checkpoint consumer groups don't preserve per-key order across segment splits and merges [pulsar]
merlimat opened a new issue, #26843:
URL: https://github.com/apache/pulsar/issues/26843
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### User environment
Pulsar 5.0.0 and `master`; V5 Java client on a scalable (`topic://`) topic.
### Issue Description
A V5 `CheckpointConsumer` that joins a consumer group (`.consumerGroup(...)`) does not preserve per-key
ordering across a split or merge of the topic's segments. After a split, the controller can assign the
sealed parent to one member and a child to another, and nothing makes the child's reader wait until the
parent has been read to its end. A key's newer messages from the child can therefore be delivered before
its older messages from the parent. Reproduced on `master` with one grouped member reading a backlog
across a split: `key-0-post-0` was delivered before `key-0-pre-217`.
The client specification makes the checkpoint consumer an ordered consumer
(`spec/scalable-topics/client-api.md` §4, "per-key, ordered"; §3.2 requires per-key order to be preserved
across segment split/merge) and does not exempt grouped consumers.
The ungrouped checkpoint consumer is fixed in #26837: it reads a segment only after its parents have been
read to their end. A group cannot do this locally, because a parent may be assigned to another member, and
the controller has no signal for it:
- For a stream subscription, the controller holds the children until the parent's broker-side cursor
reports no backlog. Checkpoint members read through Readers and keep their positions client-side, so
there is no cursor to poll; #25642 skipped CHECKPOINT subscriptions for that reason.
- A member can also rewind by restoring an older checkpoint, so "fully read" is not stable even if a member
reported it.
### Possible directions
Both change the normative client specification, so either needs a PIP:
1. **Protocol:** members report fully read sealed segments to the controller, which then holds the
children for a checkpoint group as it does for a stream subscription. This also needs a rule for members
that restore an older checkpoint.
2. **Relax the specification for grouped checkpoint consumers:** per-key order holds within a segment, and
groups are documented as not preserving it across splits and merges.
Until then, #26837 documents the limitation on `CheckpointConsumerBuilder.consumerGroup`.
### Error messages
```text
None: messages are delivered out of per-key order, without an error.
```
### Reproducing the issue
1. Create a scalable topic with one segment and a grouped `CheckpointConsumer` starting from
`Checkpoint.earliest()`.
2. Publish a keyed backlog larger than the consumer prefetches (e.g. 16 keys x 250 messages).
3. Split the segment, publish more messages for the same keys, then receive everything.
4. Some keys receive post-split messages before their remaining pre-split ones.
### Additional information
Related: #26836 (checkpoint groups never share a segment), #26842 (GC prunes ancestors first).
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] V5 checkpoint consumer groups don't preserve per-key order across segment splits and merges [pulsar]
Posted by "Asthenia0412 (via GitHub)" <gi...@apache.org>.
Asthenia0412 commented on issue #26843:
URL: https://github.com/apache/pulsar/issues/26843#issuecomment-6151925980
Hi! I'd like to contribute a fix for the checkpoint consumer group ordering across segment splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org