You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/10/20 16:35:16 UTC
[I] [Bug] V5 producer can store a message twice when it re-sends an ambiguously failed send after a segment is sealed [pulsar]
lhotari opened a new issue, #27019:
URL: https://github.com/apache/pulsar/issues/27019
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
- Broker: master (5.0.0-SNAPSHOT), scalable topics
- Client: Java V5 client (`pulsar-client-v5`), master
### Issue Description
When a split or merge seals a segment of a scalable topic, the V5 producer re-sends the sends that failed on the sealed segment to its successor segments (`ScalableTopicProducer#onSendFailure` parks them, `#releaseSends` re-sends them). This is only correct if every send that failed on the sealed segment was *not* persisted there. Otherwise the message is stored twice, once in the sealed segment and once in a successor, and every consumer receives it twice. Broker-side deduplication does not help: it works per topic, and the successor is a different topic (`segment://...`) with a different v4 producer.
The managed-ledger fix in #27018 closes the main way this happened: a terminated topic failed adds before the earlier adds in flight had completed, so the broker reported persisted messages as failed. That made `V5ProducerLayoutChangeOrderingTest` flaky with hundreds of duplicates. The V5 producer still treats a failure on a sealed segment as "not persisted", though, and a failure can still be ambiguous when the receipt is lost rather than the write:
1. The connection to the broker drops (or the broker restarts) while sends are in flight, and the segment is sealed before the v4 segment producer reconnects. The reconnect fails with `TopicTerminatedError`, and `ProducerImpl` fails all pending messages with `TopicTerminatedException` (`ProducerImpl` around L2307), including ones the broker had persisted but whose receipts were lost with the connection.
2. The layout update that drops the sealed segment arrives while its v4 producer is disconnected and still has pending messages. `ScalableTopicProducer` closes that producer ("Closing producer for sealed segment"), and the messages fail with `AlreadyClosedException`, whether or not they were persisted.
In both cases the V5 producer re-sends a message that may already be stored in the sealed segment, so the sealed segment and a successor can both hold it.
Expected: a send re-sent after a layout change is stored exactly once, the same as a v4 producer with deduplication enabled reconnecting to the same topic.
### Error messages
```text
org.opentest4j.AssertionFailedError: [messages received]
expected: 5860
but was: 6561
at org.apache.pulsar.client.impl.v5.V5ProducerLayoutChangeOrderingTest.produceAndVerifyPerKeyOrder(V5ProducerLayoutChangeOrderingTest.java:161)
```
(This CI failure, from https://github.com/apache/pulsar/actions/runs/39441347721/job/118707996775, was caused by the broker-side ordering bug fixed in #27018. It shows what the remaining ambiguous cases can lead to.)
### Reproducing the issue
There is no deterministic reproduction for the remaining cases yet. A test would need to:
1. Produce async to a scalable topic with a V5 producer.
2. Drop the client connection (or hold the send receipts) after the broker persisted some sends of a segment.
3. Split that segment before the segment producer reconnects.
4. Read the topic to the end and check that no message is stored twice.
### Additional information
Possible directions:
- Before re-sending the parked sends of a sealed segment, find out what that segment actually stored. The broker knows the highest sequence id it persisted for the producer name on that segment (the deduplication state), but producer creation on a terminated topic fails before returning it. A new way to ask for it, or a reader on the tail of the sealed segment, would let the V5 producer drop the sends that were already persisted, and complete their futures as successful.
- Or deduplicate across a segment's lineage, so that a successor rejects a sequence id that its parent already stored for the same producer.
Either needs a design discussion, possibly a PIP, since it touches the protocol or the scalable topic metadata.
Related: #26820 (per-key order of V5 async sends across layout changes).
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] V5 producer can store a message twice when it re-sends an ambiguously failed send after a segment is sealed [pulsar]
Posted by "akarazhev (via GitHub)" <gi...@apache.org>.
akarazhev commented on issue #27019:
URL: https://github.com/apache/pulsar/issues/27019#issuecomment-6317931666
I can reproduce this every time on current master (931999f, which already includes #27018).
The test below stops the client from reading the producer's connection, sends 50 messages and waits until they're stored, then splits the segment and drops the connection.
All 50 end up stored twice, four runs out of four, and both cases from your description show up in the log.
I can open a PR with the test disabled until there's a fix, if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] V5 producer can store a message twice when it re-sends an ambiguously failed send after a segment is sealed [pulsar]
Posted by "akarazhev (via GitHub)" <gi...@apache.org>.
akarazhev commented on issue #27019:
URL: https://github.com/apache/pulsar/issues/27019#issuecomment-6335309560
Got it, thanks. I mentioned #27018 only because the test runs on top of it.
When your dedup changes for V5 are up, I'm happy to run the test against them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org