You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "poorbarcode (via GitHub)" <gi...@apache.org> on 2023/02/07 04:15:02 UTC
[GitHub] [pulsar] poorbarcode opened a new pull request, #19444: [fix] [ml] topic load fail by ledger lost
poorbarcode opened a new pull request, #19444:
URL: https://github.com/apache/pulsar/pull/19444
### Motivation
Unstable ZK connections can cause these conditions to occur:
1. add entry timeout(because the connections of BK to ZK is also unstable)
1. write ZK fail when ledger info update
1. delete ledger is delayed(because delete meta is delayed)
1. and so on
When cases `1,2,3` occur at the same time, a ledger that is in use may be deleted. for example:
- add entry(`3:0`) timeout(eventually successful)
- the field `lastConfirmedPosition` of the current ledger is still `-1`
- the ledger info of ML is `[3]` now.
- switch ledger caused by add entry fail, but write ZK is also failure.
- delete ledger `3` async (because `lastConfirmedPosition == -1`), and the operation is delayed.
- topic reload caused by lost ZK connection, close old ML and create new
- execute `ml.recover`
- ledger `3` is not removed during recovery because it contains data
- create new ledger `4`
- the ledger info of ML is `[3,4]` now
- the operation of `delete ledger 3` finished.
- an error occurs: the deleted ledger `3` is still in use, and the topic load will fail by the ledger is lost.
You can reproduce this issue by `testLedgerInfoMetaCorrectIfAddEntryTimeOut`.
----
Below is a log of when an error occurs in our online environment:
##### 1. Topic load is fail
```
2022-12-02T22:33:44.115106032-05:00 stdout F 2022-12-03T03:33:44,115+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [xes/etl/persistent/stu_order][pulsar.dedup] read failed from ledger at position:21347:0
2022-12-02T22:33:44.115018833-05:00 stdout F 2022-12-03T03:33:44,114+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Error opening ledger for reading at position 21347:0 - org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
2022-12-02T22:33:44.454765118-05:00 stdout F 2022-12-03T03:33:44,454+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Error opening ledger for reading at position 21347:0 - org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
2022-12-02T22:33:44.454865574-05:00 stdout F 2022-12-03T03:33:44,454+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [xes/etl/persistent/stu_order][pulsar.dedup] read failed from ledger at position:21347:0
2022-12-02T22:33:44.957770974-05:00 stdout F 2022-12-03T03:33:44,957+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Error opening ledger for reading at position 21347:0 - org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
2022-12-02T22:33:44.957880879-05:00 stdout F 2022-12-03T03:33:44,957+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [xes/etl/persistent/stu_order][pulsar.dedup] read failed from ledger at position:21347:0
2022-12-02T22:33:45.384780673-05:00 stdout F 2022-12-03T03:33:45,384+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Error opening ledger for reading at position 21347:0 - org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
2022-12-02T22:33:45.384874832-05:00 stdout F 2022-12-03T03:33:45,384+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.mledger.impl.OpReadEntry - [xes/etl/persistent/stu_order][pulsar.dedup] read failed from ledger at position:21347:0
```
##### 2. The connection of ZK is unstable
```
# pulsar-broker-4
2022-12-02T22:23:42.659683926-05:00 stdout F 2022-12-03T03:23:42,659+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: ConnectionLost
# pulsar-broker-4
2022-12-02T22:23:42.660145113-05:00 stdout F 2022-12-03T03:23:42,659+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: ConnectionLost
# pulsar-broker-0
2022-12-02T22:23:48.738214552-05:00 stdout F 2022-12-03T03:23:48,738+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: ConnectionLost
# pulsar-broker-0
2022-12-02T22:23:48.73816115-05:00 stdout F 2022-12-03T03:23:48,738+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: ConnectionLost
# pulsar-broker-4
2022-12-02T22:24:08.660979585-05:00 stdout F 2022-12-03T03:24:08,660+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: SessionLost
# pulsar-broker-4
2022-12-02T22:24:08.660989559-05:00 stdout F 2022-12-03T03:24:08,660+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: SessionLost
# pulsar-broker-0
2022-12-02T22:24:14.703831204-05:00 stdout F 2022-12-03T03:24:14,703+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: SessionLost
# pulsar-broker-0
2022-12-02T22:24:14.7038165-05:00 stdout F 2022-12-03T03:24:14,703+0000 [metadata-store-zk-session-watcher-11-1] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: SessionLost
# pulsar-broker-4
2022-12-02T22:25:16.667165767-05:00 stdout F 2022-12-03T03:25:16,666+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: Reconnected
# pulsar-broker-4
2022-12-02T22:25:16.667042947-05:00 stdout F 2022-12-03T03:25:16,666+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: Reconnected
# pulsar-broker-4
2022-12-02T22:25:16.681716421-05:00 stdout F 2022-12-03T03:25:16,681+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: SessionReestablished
# pulsar-broker-4
2022-12-02T22:25:16.681441416-05:00 stdout F 2022-12-03T03:25:16,681+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: SessionReestablished
# pulsar-broker-4
2022-12-02T22:25:16.761244318-05:00 stdout F 2022-12-03T03:25:16,760+0000 [metadata-store-coordination-service-12-1] WARN org.apache.pulsar.metadata.coordination.impl.LockManagerImpl - Failure when processing session event
# pulsar-broker-0
2022-12-02T22:33:24.741380081-05:00 stdout F 2022-12-03T03:33:24,741+0000 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: Reconnected
# pulsar-broker-0
2022-12-02T22:33:24.74137549-05:00 stdout F 2022-12-03T03:33:24,741+0000 [main-EventThread] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: Reconnected
# pulsar-broker-0
2022-12-02T22:33:24.755746638-05:00 stdout F 2022-12-03T03:33:24,754+0000 [main-EventThread] INFO org.apache.pulsar.broker.PulsarService - Received metadata service session event: SessionReestablished
# pulsar-broker-0
2022-12-02T22:33:24.755772892-05:00 stdout F 2022-12-03T03:33:24,754+0000 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - Received MetadataStore session event: SessionReestablished
```
##### 3.The Topic switches brokers frequently
```
# pulsar-broker-4
2022-12-02T22:21:01.15839105-05:00 stdout F 2022-12-03T03:21:01,157+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:21:01.679585396-05:00 stdout F 2022-12-03T03:21:01,679+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-0
2022-12-02T22:21:35.923492915-05:00 stdout F 2022-12-03T03:21:35,923+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:21:36.252018422-05:00 stdout F 2022-12-03T03:21:36,251+0000 [pulsar-io-4-3] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-4
2022-12-02T22:23:11.159764484-05:00 stdout F 2022-12-03T03:23:11,159+0000 [pulsar-io-4-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:23:11.333602486-05:00 stdout F 2022-12-03T03:23:11,333+0000 [pulsar-io-4-1] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-0
2022-12-02T22:23:12.962155931-05:00 stdout F 2022-12-03T03:23:12,961+0000 [pulsar-io-4-3] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:23:13.20731387-05:00 stdout F 2022-12-03T03:23:13,207+0000 [pulsar-io-4-3] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-4
2022-12-02T22:26:35.030681434-05:00 stdout F 2022-12-03T03:26:35,030+0000 [pulsar-io-4-2] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:27:01.794766318-05:00 stdout F 2022-12-03T03:27:01,794+0000 [pulsar-io-4-2] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-0
2022-12-02T22:33:24.867254146-05:00 stdout F 2022-12-03T03:33:24,866+0000 [pulsar-io-4-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:33:25.097914184-05:00 stdout F 2022-12-03T03:33:25,097+0000 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
# pulsar-broker-4
2022-12-02T22:36:01.570683778-05:00 stdout F 2022-12-03T03:36:01,570+0000 [pulsar-io-4-4] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://xes/etl/stu_order] Failed to close managed ledger, proceeding anyway.
2022-12-02T22:36:01.755647415-05:00 stdout F 2022-12-03T03:36:01,755+0000 [pulsar-io-4-4] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger xes/etl/persistent/stu_order
2022-12-02T22:38:47.212421358-05:00 stdout F 2022-12-03T03:38:47,212+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/default/persistent/__sla
```
##### 4. Ledger-21347's life cycle
```
2022-12-02T22:23:13.250490551-05:00 stdout F 2022-12-03T03:23:13,250+0000 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-1.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 21347
2022-12-02T22:23:13.250647455-05:00 stdout F 2022-12-03T03:23:13,250+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Created ledger 21347
2022-12-02T22:24:14.296700424-05:00 stdout F 2022-12-03T03:24:14,296+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (21347, 0): Bookie operation timeout
2022-12-02T22:27:01.888372507-05:00 stdout F 2022-12-03T03:27:01,888+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.client.ReadOnlyLedgerHandle - Closing recovered ledger 21347 at entry 0
2022-12-02T22:33:24.742114702-05:00 stdout F 2022-12-03T03:33:24,742+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.client.LedgerHandle - New Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 21347
2022-12-02T22:33:24.743599829-05:00 stdout F 2022-12-03T03:33:24,743+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L21347 E0
2022-12-02T22:33:24.743596141-05:00 stdout F 2022-12-03T03:33:24,743+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 21347 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read
2022-12-02T22:33:24.743589628-05:00 stdout F 2022-12-03T03:33:24,743+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN org.apache.bookkeeper.client.PendingAddOp - Fencing exception on write: L21347 E0 on pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181
2022-12-02T22:33:24.772910842-05:00 stdout F 2022-12-03T03:33:24,772+0000 [main-EventThread] INFO org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=21347,loopId=31b528bd) conflict writing metadata to store, update local value and try again
2022-12-02T22:33:24.787555553-05:00 stdout F 2022-12-03T03:33:24,787+0000 [main-EventThread] ERROR org.apache.bookkeeper.client.LedgerHandle - Metadata conflict when closing ledger 21347. Another client may have recovered the ledger while there were writes outstanding. (local lastEntry:-1 length:0) (metadata lastEntry:0 length:815)
2022-12-02T22:33:24.788055839-05:00 stdout F 2022-12-03T03:33:24,787+0000 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=21347,loopId=31b528bd) Exception updating
2022-12-02T22:33:24.802360201-05:00 stdout F 2022-12-03T03:33:24,802+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xes/etl/persistent/stu_order] Delete complete for empty ledger 2134. rc=0
```
### Modifications
Only ledgers removed from the meta of ledger info can be deleted from the BK.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository:
- https://github.com/poorbarcode/pulsar/pull/65
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1098365963
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1706,19 +1710,10 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
if (log.isDebugEnabled()) {
log.debug("[{}] Ledger has been closed id={} entries={}", name, lh.getId(), entriesInLedger);
}
- if (entriesInLedger > 0) {
- LedgerInfo info = LedgerInfo.newBuilder().setLedgerId(lh.getId()).setEntries(entriesInLedger)
- .setSize(lh.getLength()).setTimestamp(clock.millis()).build();
- ledgers.put(lh.getId(), info);
- } else {
- // The last ledger was empty, so we can discard it
- ledgers.remove(lh.getId());
- mbean.startDataLedgerDeleteOp();
- bookKeeper.asyncDeleteLedger(lh.getId(), (rc, ctx) -> {
- mbean.endDataLedgerDeleteOp();
- log.info("[{}] Delete complete for empty ledger {}. rc={}", name, lh.getId(), rc);
- }, null);
Review Comment:
> Why we need remove these lines?
This method is part of the ledger switch flow, the whole ledger switch flow is like this:
1. close the current ledger (Here it is).
2. create a new ledger
3. update ledgers to ZK
If we delete `current ledger` from BK here, and update ledgers to ZK fail, it will cause an issue: this ledger is still used but has been deleted, so we can not delete this ledger here.
if we remove the current ledger from variable `ledgers` only, then after updating ledger info to ZK, we can not get the ledgerId which should be deleted.
----
<strong>(Highlight)</strong>I think you are right, there has another way to fix it:
- remove the current ledger from variable `ledgers` only
- update ledger info to ZK
- get ledgerId by variable `currentLedger`, and delete it from BK
Already edited this code, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1436220260
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1099429975
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1650,8 +1656,15 @@ void createNewOpAddEntryForNewLedger() {
} while (existsOp != null && --pendingSize > 0);
}
- protected synchronized void updateLedgersIdsComplete() {
+ protected synchronized void updateLedgersIdsComplete(@Nullable LedgerHandle originalCurrentLedger) {
STATE_UPDATER.set(this, State.LedgerOpened);
+ // Delete original "currentLedger" if it has been removed from "ledgers".
+ if (originalCurrentLedger != null && !ledgers.containsKey(originalCurrentLedger.getId())){
+ bookKeeper.asyncDeleteLedger(originalCurrentLedger.getId(), (rc, ctx) -> {
+ mbean.endDataLedgerDeleteOp();
+ log.info("[{}] Delete complete for empty ledger {}. rc={}", name, originalCurrentLedger.getId(), rc);
Review Comment:
Why do we call this an empty ledger? Should we give a warn log when delete fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1432592134
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1100771062
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -475,6 +476,7 @@ protected synchronized void initializeBookKeeper(final ManagedLedgerInitializeLe
}
// Calculate total entries and size
+ final List<Long> emptyLedgersToBeDeleted = new ArrayList<>();
Review Comment:
Is there a reason we are not using a thread safe list here? I think we might need one, though I have not read through all of this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1433178582
rebase master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1098288786
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1706,19 +1710,10 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
if (log.isDebugEnabled()) {
log.debug("[{}] Ledger has been closed id={} entries={}", name, lh.getId(), entriesInLedger);
}
- if (entriesInLedger > 0) {
- LedgerInfo info = LedgerInfo.newBuilder().setLedgerId(lh.getId()).setEntries(entriesInLedger)
- .setSize(lh.getLength()).setTimestamp(clock.millis()).build();
- ledgers.put(lh.getId(), info);
- } else {
- // The last ledger was empty, so we can discard it
- ledgers.remove(lh.getId());
- mbean.startDataLedgerDeleteOp();
- bookKeeper.asyncDeleteLedger(lh.getId(), (rc, ctx) -> {
- mbean.endDataLedgerDeleteOp();
- log.info("[{}] Delete complete for empty ledger {}. rc={}", name, lh.getId(), rc);
- }, null);
Review Comment:
Why we need remove these lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1098365963
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1706,19 +1710,10 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
if (log.isDebugEnabled()) {
log.debug("[{}] Ledger has been closed id={} entries={}", name, lh.getId(), entriesInLedger);
}
- if (entriesInLedger > 0) {
- LedgerInfo info = LedgerInfo.newBuilder().setLedgerId(lh.getId()).setEntries(entriesInLedger)
- .setSize(lh.getLength()).setTimestamp(clock.millis()).build();
- ledgers.put(lh.getId(), info);
- } else {
- // The last ledger was empty, so we can discard it
- ledgers.remove(lh.getId());
- mbean.startDataLedgerDeleteOp();
- bookKeeper.asyncDeleteLedger(lh.getId(), (rc, ctx) -> {
- mbean.endDataLedgerDeleteOp();
- log.info("[{}] Delete complete for empty ledger {}. rc={}", name, lh.getId(), rc);
- }, null);
Review Comment:
> Why we need remove these lines?
This method is part of the ledger switch flow, the whole ledger switch flow is like this:
1. close the current ledger (Here it is).
2. create a new ledger
3. update ledgers to ZK
If we delete `current ledger` from BK here, and update ledgers to ZK fail, it will cause an issue: this ledger is still used but has been deleted, so we can not delete this ledger here.
if we remove the current ledger from variable `ledgers` only, then after updating ledger info to ZK, we can not get the ledgerId which should be deleted.
----
<strong>(Highlight)</strong>I think you are right, there has another way to fix it:
- remove the current ledger from variable `ledgers` only
- updating ledger info to ZK
- get ledgerId by variable `currentLedger`, and delete it from BK
Already edited this code, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1100937300
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -475,6 +476,7 @@ protected synchronized void initializeBookKeeper(final ManagedLedgerInitializeLe
}
// Calculate total entries and size
+ final List<Long> emptyLedgersToBeDeleted = new ArrayList<>();
Review Comment:
> Is there a reason we are not using a thread-safe list here?
It will not be read or written by multiple threads. And It has been changed to a thread-safe list to avoid future problems when others modify the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode closed pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode closed pull request #19444: [fix] [ml] topic load fail by ledger lost
URL: https://github.com/apache/pulsar/pull/19444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1433422876
I have written a PR to fix this flaky test `ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors` :
- https://github.com/apache/pulsar/pull/19545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1098365963
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1706,19 +1710,10 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
if (log.isDebugEnabled()) {
log.debug("[{}] Ledger has been closed id={} entries={}", name, lh.getId(), entriesInLedger);
}
- if (entriesInLedger > 0) {
- LedgerInfo info = LedgerInfo.newBuilder().setLedgerId(lh.getId()).setEntries(entriesInLedger)
- .setSize(lh.getLength()).setTimestamp(clock.millis()).build();
- ledgers.put(lh.getId(), info);
- } else {
- // The last ledger was empty, so we can discard it
- ledgers.remove(lh.getId());
- mbean.startDataLedgerDeleteOp();
- bookKeeper.asyncDeleteLedger(lh.getId(), (rc, ctx) -> {
- mbean.endDataLedgerDeleteOp();
- log.info("[{}] Delete complete for empty ledger {}. rc={}", name, lh.getId(), rc);
- }, null);
Review Comment:
> Why we need remove these lines?
This method is part of the ledger switch flow, the whole ledger switch flow is like this:
1. close the current ledger (Here it is).
2. create a new ledger
3. update ledgers to ZK
If we delete `current ledger` from BK here, and update ledgers to ZK fail, it will cause an issue: this ledger is still used but has been deleted, so we can not delete this ledger here.
if we remove the current ledger from variable `ledgers` only, then after updating ledger info to ZK, we can not get the ledgerId which should be deleted.
----
I think you are right, there has another way to fix it:
- remove the current ledger from variable `ledgers` only
- updating ledger info to ZK
- get ledgerId by variable `currentLedger`, and delete it from BK
Already edited this code, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1431629157
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1099429975
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1650,8 +1656,15 @@ void createNewOpAddEntryForNewLedger() {
} while (existsOp != null && --pendingSize > 0);
}
- protected synchronized void updateLedgersIdsComplete() {
+ protected synchronized void updateLedgersIdsComplete(@Nullable LedgerHandle originalCurrentLedger) {
STATE_UPDATER.set(this, State.LedgerOpened);
+ // Delete original "currentLedger" if it has been removed from "ledgers".
+ if (originalCurrentLedger != null && !ledgers.containsKey(originalCurrentLedger.getId())){
+ bookKeeper.asyncDeleteLedger(originalCurrentLedger.getId(), (rc, ctx) -> {
+ mbean.endDataLedgerDeleteOp();
+ log.info("[{}] Delete complete for empty ledger {}. rc={}", name, originalCurrentLedger.getId(), rc);
Review Comment:
Why do we call this an empty ledger? Should we give a warn log when delete fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1424204367
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode merged pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode merged PR #19444:
URL: https://github.com/apache/pulsar/pull/19444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1438125612
rebase master after the flaky test fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#discussion_r1098365963
##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -1706,19 +1710,10 @@ synchronized void ledgerClosed(final LedgerHandle lh) {
if (log.isDebugEnabled()) {
log.debug("[{}] Ledger has been closed id={} entries={}", name, lh.getId(), entriesInLedger);
}
- if (entriesInLedger > 0) {
- LedgerInfo info = LedgerInfo.newBuilder().setLedgerId(lh.getId()).setEntries(entriesInLedger)
- .setSize(lh.getLength()).setTimestamp(clock.millis()).build();
- ledgers.put(lh.getId(), info);
- } else {
- // The last ledger was empty, so we can discard it
- ledgers.remove(lh.getId());
- mbean.startDataLedgerDeleteOp();
- bookKeeper.asyncDeleteLedger(lh.getId(), (rc, ctx) -> {
- mbean.endDataLedgerDeleteOp();
- log.info("[{}] Delete complete for empty ledger {}. rc={}", name, lh.getId(), rc);
- }, null);
Review Comment:
> Why we need remove these lines?
This method is part of the ledger switch flow, the whole ledger switch flow is like this:
1. close the current ledger (Here it is).
2. create a new ledger
3. update ledgers to ZK
If we delete `current ledger` from BK here, and update ledgers to ZK fail, it will cause an issue: this ledger is still used but has been deleted, so we can not delete this ledger here.
if we remove the current ledger from variable `ledgers` only, then after updating ledger info to ZK, we can not get the ledgerId which should be deleted.
### new implementation
- remove the current ledger from variable `ledgers` only
- updating ledger info to ZK
- get ledgerId by variable `currentLedger`, and delete it from BK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1421941307
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed pull request #19444: [fix] [ml] topic load fail by ledger lost
URL: https://github.com/apache/pulsar/pull/19444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #19444: [fix] [ml] topic load fail by ledger lost
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on PR #19444:
URL: https://github.com/apache/pulsar/pull/19444#issuecomment-1438484857
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org