You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/06/16 03:12:57 UTC
[GitHub] [bookkeeper] Nicklee007 opened a new pull request, #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Nicklee007 opened a new pull request, #3339:
URL: https://github.com/apache/bookkeeper/pull/3339
Master Issue: #3338
### Motivation
As the Decommissioning bookie case， always change the bookie status to readonly firstly, and then wait some data expired, but always it has some ledgers (about 100+ -- 300+) legacy not be cleaned and the leaved ledgers only has little data , when we running `bin/bookkeeper shell decommissionbookie -bookieid ` to decommission the bookie , we always pending on `waitForLedgersToBeReplicated()` about 10 min and have not any log print, but we could find the znode /ledgers/underreplication/ledgers cleaned only few seconds and then the ledgers be rereplicate completed, we find the sleep time is defined as the `Min(ledgers.size() * sleepTimePerLedger(10s) , maxSleepTimeInBetweenChecks(10 min))`, in the way , the both time always wait too long and has not any print will let user confused。
### Changes
In the bookie has many data to rereplicate case , we need the backoff policy to protect zk server, so
1. To look the `/ledgers/underreplication/ledgers ` and ` /ledgers/underreplication/locks` every 10 sec, help us check if the ledgers replicate completed.
2. To avoid the auditor is running as CheckAllLedgers or other time-consuming operation when we trigger the audit bookie, then `/ledgers/underreplication/ledgers ` and ` /ledgers/underreplication/locks` will empty to long time , we need a back off policy to avoid frequent request zk to `validateBookieIsNotPartOfEnsemble`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] Nicklee007 commented on pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#issuecomment-1174772916
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] Nicklee007 commented on pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#issuecomment-1167421576
@eolivelli @dlg99 Could you help to check the logic is correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] StevenLuMT commented on pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#issuecomment-1225341074
fix old workflow,please see #3455 for detail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] decommissionBookie waiting for ledgers to be replicated back off policy [bookkeeper]
Posted by "zymap (via GitHub)" <gi...@apache.org>.
zymap closed pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
URL: https://github.com/apache/bookkeeper/pull/3339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] Nicklee007 commented on pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Posted by GitBox <gi...@apache.org>.
Nicklee007 commented on PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#issuecomment-1169483308
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3339: [improve] decommissionBookie waiting for ledgers to be replicated back off policy
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#discussion_r929505521
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java:
##########
@@ -1622,13 +1622,33 @@ private void waitForLedgersToBeReplicated(Collection<Long> ledgers, BookieId thi
LedgerManager ledgerManager) throws InterruptedException, TimeoutException {
int maxSleepTimeInBetweenChecks = 10 * 60 * 1000; // 10 minutes
int sleepTimePerLedger = 10 * 1000; // 10 secs
+ int rereplicateBackoffMs = 10 * 1000; // 10 secs
+ int backOffCount = 0;
Predicate<Long> validateBookieIsNotPartOfEnsemble = ledgerId -> !areEntriesOfLedgerStoredInTheBookie(ledgerId,
thisBookieAddress, ledgerManager);
while (!ledgers.isEmpty()) {
LOG.info("Count of Ledgers which need to be rereplicated: {}", ledgers.size());
int sleepTimeForThisCheck = (long) ledgers.size() * sleepTimePerLedger > maxSleepTimeInBetweenChecks
? maxSleepTimeInBetweenChecks : ledgers.size() * sleepTimePerLedger;
- Thread.sleep(sleepTimeForThisCheck);
+ long nextValidateTime = System.currentTimeMillis() + sleepTimeForThisCheck;
Review Comment:
Could you use a backoff policy to simplify the code? You can refer to: https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/Backoff.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] decommissionBookie waiting for ledgers to be replicated back off policy [bookkeeper]
Posted by "zymap (via GitHub)" <gi...@apache.org>.
zymap commented on PR #3339:
URL: https://github.com/apache/bookkeeper/pull/3339#issuecomment-1837767759
Close this since it is open for a long time without any updates. Feel free to reopen it if you want to continue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org