You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "nodece (via GitHub)" <gi...@apache.org> on 2026/08/03 05:03:04 UTC
[PR] [improve][ml] Per-msgLedger individual ack persistence (DE + CM model) [pulsar]
nodece opened a new pull request, #26266:
URL: https://github.com/apache/pulsar/pull/26266
## Motivation
The current cursor persistence writes the entire subscription's individual ack state as a single `PositionInfo` entry per flush. This causes:
1. **Write amplification** — the full ack state is re-serialized every flush even when only one message-ledger changed
2. **Size cliff** — entries exceeding BookKeeper's 5MB limit are silently truncated by `managedLedgerMaxUnackedRangesToPersist`, losing acknowledged messages
3. **Recovery memory pressure** — the full state is loaded into memory at once on broker failover
## Modifications
Introduce a per-msgLedger **Data Entry (DE)** + **Checkpoint Marker (CM)** model, controlled by `persistentUnackedRangesWithPerLedgerEntryEnabled` (default `false`).
### Architecture
```
AckPersistence (interface)
├── LegacyAckPersistence — single PositionInfo per flush (unchanged)
└── IndividualAckStatePersistence — DE + CM model
└── CursorWal — BK I/O (transparent chunking)
BitmapAckState (in-memory)
├── PositionRangeSet — per-ledger RoaringBitmap + dirty tracking
└── batch ack tracking
```
### Wire format (per-ledger-entry mode)
Each flush writes:
1. One **DE** per dirty message-ledger: RoaringBitmap ack state + batch ack indexes
2. One **CM**: mark-delete position + properties + DE index (`msgLedgerId → entry position`)
Recovery reads the last entry (always a CM), parallel-fetches all referenced DEs, and merges into a synthetic `PositionInfo`.
### Key features
- **Backward compatible**: feature flag default off, legacy path unchanged
- **Cross-ledger DE references**: old cursor ledgers are retained for CM references
- **Old cursor ledger GC**: based on `oldestReferencedCursorLedgerId`, cleaned up on mark-delete advance and rollover
- **ZK hint**: CM location piggybacked onto existing ZK writes for faster recovery
- **Transparent chunking**: `CursorWal` splits oversized entries via `CursorStateChunk` envelopes
### Proto changes
- `CursorStateChunk`: simplified to 3 fields (`partIndex`, `partCount`, `payload`)
- `LongListMap.bitmap`: RoaringBitmap native serialization field (field 3)
- `ManagedCursorInfo.lastCmCursorLedgerId/lastCmEntryId`: ZK recovery hint (fields 9, 10)
### Config changes
- `persistentUnackedRangesWithPerLedgerEntryEnabled` (default `false`): enables DE+CM model
- `persistentUnackedRangesMaxEntrySize` default raised from 1MB to `maxMessageSize` (5MB)
## Test plan
- [x] `testDeCmIndividualAckPersistAndRecover` — individual ack persist + recover via DE+CM
- [x] `testDeCmBatchAckPersistAndRecover` — batch ack entries stored inside DEs are correctly recovered
- [x] `testDeCmMultiMsgLedgerWithCursorRollover` — ack across multiple message-ledgers + recovery
- [x] `testDeCmCursorLedgerRolloverAndGC` — cursor ledger rollover with cross-ledger DE references + GC
- [x] `testMultiEntryIndividualAckStatePersistAndRecover` — bitmap serialization format via legacy path
- [x] `testLegacyLongArrayWithSingleAckHole` — legacy long-array format backward compatibility
- [x] `CursorWalTest` — chunked entry assembly, partial write recovery, partIndex continuity, boundary
- [x] Full `ManagedCursorTest` suite (84 tests) passes with zero regressions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][ml] Per-msgLedger individual ack persistence (DE + CM model) [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece closed pull request #26266: [improve][ml] Per-msgLedger individual ack persistence (DE + CM model)
URL: https://github.com/apache/pulsar/pull/26266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org