You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "codelipenghui (via GitHub)" <gi...@apache.org> on 2025/12/17 22:11:00 UTC
[PR] [fix][ml] Fix cursor backlog size to account for individual acks [pulsar]
codelipenghui opened a new pull request, #25089:
URL: https://github.com/apache/pulsar/pull/25089
Fixes the issue where cursor.getEstimatedSizeSinceMarkDeletePosition() would not subtract the size of individually acknowledged messages.
Problem:
- When messages are acknowledged out of order using asyncDelete(), the backlog count correctly decreases
- But the backlog size remains inflated because individualDeletedMessages are not accounted for
Example:
- 5 messages (500 bytes total)
- Individual ack of 3 messages
- Before fix: backlog size = 500 (incorrect)
- After fix: backlog size = 200 (correct)
Fix:
- Update ManagedCursorImpl.getEstimatedSizeSinceMarkDeletePosition() to calculate and subtract size of individual deleted entries
- Uses average entry size to estimate deleted size
- Thread-safe with proper locking
Unit test:
- testEstimatedSizeWithIndividualAcks() verifies the fix
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][ml] Fix cursor backlog size to account for individual acks [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #25089:
URL: https://github.com/apache/pulsar/pull/25089#discussion_r2642242646
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java:
##########
@@ -1282,7 +1282,69 @@ public int getNonContiguousDeletedMessagesRangeSerializedSize() {
@Override
public long getEstimatedSizeSinceMarkDeletePosition() {
- return ledger.estimateBacklogFromPosition(markDeletePosition);
+ long totalSize = ledger.estimateBacklogFromPosition(markDeletePosition);
+
+ // Need to subtract size of individual deleted messages
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Calculating backlog size for cursor {} from position {}, totalSize: {}",
+ ledger.getName(), name, markDeletePosition, totalSize);
+ }
+
+ // Get count of individually deleted entries in the backlog range
+ long deletedCount = 0;
+ lock.readLock().lock();
+ try {
+ Range<Position> backlogRange = Range.openClosed(markDeletePosition, ledger.getLastPosition());
+
+ if (getConfig().isUnackedRangesOpenCacheSetEnabled()) {
+ deletedCount = individualDeletedMessages.cardinality(
+ backlogRange.lowerEndpoint().getLedgerId(), backlogRange.lowerEndpoint().getEntryId(),
+ backlogRange.upperEndpoint().getLedgerId(), backlogRange.upperEndpoint().getEntryId());
+ } else {
+ AtomicLong deletedCounter = new AtomicLong(0);
+ individualDeletedMessages.forEach((r) -> {
+ if (r.isConnected(backlogRange)) {
+ Range<Position> intersection = r.intersection(backlogRange);
+ long countInRange = ledger.getNumberOfEntries(intersection);
+ deletedCounter.addAndGet(countInRange);
+ }
+ return true;
+ }, recyclePositionRangeConverter);
+ deletedCount = deletedCounter.get();
+ }
+ } finally {
+ lock.readLock().unlock();
+ }
Review Comment:
This part seems to duplicate logic that currently exists in `getNumberOfEntries` method. Avoiding code duplication would be preferred. Would it be possible to extract a common method that is used in both `getEstimatedSizeSinceMarkDeletePosition` and `getNumberOfEntries`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][ml] Fix cursor backlog size to account for individual acks [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #25089:
URL: https://github.com/apache/pulsar/pull/25089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][ml] Fix cursor backlog size to account for individual acks [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on PR #25089:
URL: https://github.com/apache/pulsar/pull/25089#issuecomment-3667410996
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org