You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/27 11:25:56 UTC
[GitHub] [pulsar] wolfstudy opened a new issue, #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
wolfstudy opened a new issue, #19077:
URL: https://github.com/apache/pulsar/issues/19077
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Bookie Version：4.14.4
Broker Version： 2.9.2
OS: Linux Centos7
### Minimal reproduce step
Under the above version, we found that some EntryLogs in Bookie could not be deleted for a long time. At first, we thought that it was caused by not triggering minor GC or major GC. Then we adjusted the thresholds of minor GC and major GC, and found that the recovery effect was not very obvious. Then we scanned the problematic EntryLog file, and then obtained the list of all Ledgers in the current EntryLog, and obtained the ledger metadata corresponding to each Ledger, and then parsed out the Topic information corresponding to the Ledger, and found the following situation:
1. The ledger metadata as follows:
```
2022-12-26 20:28:06,184 - INFO - [main:ParseEntryLog@217] - ManagedLedger: LedgerID: 1174539 --- LedgerCreateTime: 1654146303273 --- LedgerState: CLOSED --- Ensembles: {0=[xxx:3181, xxx:3181, xxx:3181]} --- TopicName: pulsar-xxx/test/test2-partition-2
```
2. The topic stats internal as follows
[stats-internal.log](https://github.com/apache/pulsar/files/10307780/internal.log)
We can see that the `markDeletePosition` of all subscriptions under the current topic has been updated to the last message, indicating that all the messages in this topic have been correctly consumed and confirmed. But in fact, this ledger has not been deleted, which leads to the fact that the proportion of valid data in the EntryLog does not meet the triggering conditions of major GC, which further causes the EntryLog to exist for a long time and cannot be deleted.
Observing the topic stats internal, we can see that it is currently at a boundary position, markDeletePosition is at the position of the last message of the previous Ledger, and there is no message in the next new Ledger, so whether there is a Ledger in the boundary case If it cannot be deleted, observe that the state of the ledger is already in the CLOSED state
### What did you expect to see?
The CLOSED ledger will be deleted.
### What did you see instead?
The CLOSED ledger not be deleted.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
URL: https://github.com/apache/pulsar/issues/19077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1366405634
Found a new thread, for this case, all ledger flags have the status: NoLedger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1659449935
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by github-actions.
github-actions[bot] commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1407253915
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1365967860
Maybe it's due to the read position is still belong to the ledger `1174539`, so the trim ledger task will not check the ledger `1174539`, it only checks ledgers less than `1174539`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1366346251
> It seems that it's due to the read position still belonging to the ledger `1174539`, so the trim ledger task will not check the ledger `1174539`, it only checks ledgers less than `1174539`.
The processing that should belong to the boundary case here is inappropriate, because the calculation of entries in stats-internal starts from 1. But the position of the first message messageID in a ledger is ledgerID:entryID = 1:0. So when MarkDeletePosition is marked to 1174539:226934, it means that 226935 messages in ledger 1174539 have been Acked, because we have set retention and TTL in the background, the maximum is no more than 10 days, and this Ledger is the data of 3 months ago, so These data must be expired by TTL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 3286360470 commented on issue #19077: [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases
Posted by "3286360470 (via GitHub)" <gi...@apache.org>.
3286360470 commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-1614025653
Is this problem solved? Anyone else looking at this question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] MarkDeletePosition causes the corresponding ledger to not be deleted in borderline cases [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #19077:
URL: https://github.com/apache/pulsar/issues/19077#issuecomment-6387057674
Thanks @wolfstudy for the detailed report. This boundary case was fixed by https://github.com/apache/pulsar/pull/22034. When the slowest mark-delete position is the last entry of a closed ledger, trimming now treats the next ledger as the slowest one, so the fully acknowledged ledger can be deleted (`ManagedLedgerImpl#internalTrimLedgers` in `managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java` on master). The same PR also creates a new ledger after a rollover even when no writes are pending. The fix is in 4.0.x, 4.2.x and 5.0.x. I'm closing this. If you still see fully acknowledged ledgers that aren't deleted on a currently supported release, please reopen with the topic's stats-internal output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org