You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "lifepuzzlefun (via GitHub)" <gi...@apache.org> on 2023/09/02 09:29:07 UTC
[GitHub] [bookkeeper] lifepuzzlefun opened a new pull request, #4069: Parallel checkpoint when dbledgerStorage config multi ledgerStorageDir
lifepuzzlefun opened a new pull request, #4069:
URL: https://github.com/apache/bookkeeper/pull/4069
Descriptions of the changes in this PR:
### Motivation
Our production env config multi journal and with one ledgerStorage on nvme disk.
but the problem occurs with checkpoint which is slow. and disk have much more performance to use.
after we changed to multi ledgerStorageDir, the SyncThread will become bottleneck, which will try to checkpoint one by one for all the ledgerStorage.
### Changes
if the DbLedgerStorage is config with multi ledgerStorage Dir try to submit each dirs task to executor to speed up checkpoint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #4069: Parallel checkpoint when dbledgerStorage config multi ledgerStorageDir
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on PR #4069:
URL: https://github.com/apache/bookkeeper/pull/4069#issuecomment-1704529904
>after we changed to multi ledgerStorageDir, the SyncThread will become bottleneck
@lifepuzzlefun Could you provide more detail about the SyncThread bottleneck?
Checkpoint will triggered in two ways:
- Triggered by addEntry in SingleDirectoryDbLedgerStorage, which is executed by the thread in SingleDirectoryDbLedgerStorage
- Triggered by SyncThread every 60s by default
When high throughput writes in to bookie, the SyncThread won't be the bottleneck for the throughput because the checkpoint will be triggered by the thread in SingleDirectoryDbLedgerStorage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #4069: Parallel checkpoint when dbledgerStorage config multi ledgerStorageDir
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on code in PR #4069:
URL: https://github.com/apache/bookkeeper/pull/4069#discussion_r1314387070
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java:
##########
@@ -388,8 +404,41 @@ public void flush() throws IOException {
@Override
public void checkpoint(Checkpoint checkpoint) throws IOException {
+ if (checkpointExecutor != null) {
Review Comment:
The checkpoint is executed by SyncThread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] lifepuzzlefun closed pull request #4069: Parallel checkpoint when dbledgerStorage config multi ledgerStorageDir
Posted by "lifepuzzlefun (via GitHub)" <gi...@apache.org>.
lifepuzzlefun closed pull request #4069: Parallel checkpoint when dbledgerStorage config multi ledgerStorageDir
URL: https://github.com/apache/bookkeeper/pull/4069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org