You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/12/25 02:32:31 UTC
[GitHub] [bookkeeper] hangc0276 opened a new pull request #2963: Add throttle for rebuild entryMetadataMap
hangc0276 opened a new pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963
### Motivation
When a bookie restart, the garbageCollectorThread will rebuild entryMetadataMap from all the entry log files in ledger directory. For normal case, it will extract the EntryLogMetadata from the index in entry log file. However, if there's no index, then fallback to scanning the entry log file.
In user's production environment, the log files without index occupied 4%. The total entry log files is 80000, and the log files without index is 3000. The default entry log file size is 2GB, and the garbageCollectorThread will read 3000 * 2GB = 6TB data without speed limit, which will cause ledger disk IO util runs high for dozens of minutes and affect ledger read and write latency.
### Modification
1. Add read speed rate limiter for scanning entry log file in entryMetadataMap rebuild.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 merged pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
dlg99 merged pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1039765714
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on a change in pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#discussion_r775127967
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogger.java
##########
@@ -1067,6 +1067,20 @@ public EntryLogMetadata getEntryLogMetadata(long entryLogId) throws IOException
}
}
+ public EntryLogMetadata getEntryLogMetadata(long entryLogId, AbstractLogCompactor.Throttler throttler)
+ throws IOException {
+ // First try to extract the EntryLogMetadata from the index, if there's no index then fallback to scanning the
+ // entry log
+ try {
+ return extractEntryLogMetadataFromIndex(entryLogId);
+ } catch (Exception e) {
Review comment:
Can we catch specific exceptions here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1045994176
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1002347261
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1001280988
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1001546260
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1042782500
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1039765714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1001889476
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1039480401
@hangc0276 please rebase/resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1039778747
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] dlg99 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1039480401
@hangc0276 please rebase/resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on a change in pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#discussion_r778592419
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogger.java
##########
@@ -1055,18 +1055,32 @@ public void scanEntryLog(long entryLogId, EntryLogScanner scanner) throws IOExce
}
public EntryLogMetadata getEntryLogMetadata(long entryLogId) throws IOException {
- // First try to extract the EntryLogMetada from the index, if there's no index then fallback to scanning the
+ // First try to extract the EntryLogMetadata from the index, if there's no index then fallback to scanning the
// entry log
try {
return extractEntryLogMetadataFromIndex(entryLogId);
- } catch (Exception e) {
+ } catch (IOException e) {
LOG.info("Failed to get ledgers map index from: {}.log : {}", entryLogId, e.getMessage());
// Fall-back to scanning
return extractEntryLogMetadataByScanning(entryLogId);
}
}
+ public EntryLogMetadata getEntryLogMetadata(long entryLogId, AbstractLogCompactor.Throttler throttler)
+ throws IOException {
+ // First try to extract the EntryLogMetadata from the index, if there's no index then fallback to scanning the
+ // entry log
+ try {
+ return extractEntryLogMetadataFromIndex(entryLogId);
Review comment:
I wonder if you could add a test that injects a failure here using PowerMock and test that the bookie works even if we fall into the catch clause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2963: Add throttle for rebuild entryMetadataMap
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2963:
URL: https://github.com/apache/bookkeeper/pull/2963#issuecomment-1005300313
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org