[bookkeeper] branch master updated: Fixed journal trimming after write cache flush in DbLedgerStorage

[mm...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git


The following commit(s) were added to refs/heads/master by this push:
     new 70b2e4c  Fixed journal trimming after write cache flush in DbLedgerStorage
70b2e4c is described below

commit 70b2e4cf49d2fc4903c277d309dbe8c0bc5e4d7d
Author: Matteo Merli <mm...@apache.org>
AuthorDate: Wed Mar 14 16:32:36 2018 -0700

    Fixed journal trimming after write cache flush in DbLedgerStorage
    
    `DbLedgerStorage` uses an in-memory write cache to accumulate entries before writing into the entry log.
    
    The flush of the write cache is triggered by 2 factors:
     1. `SyncThread` issuing a checkpoint (eg: every 10sec or 1min)
     2. Write cache is full, so we need to force the flush
    
    In case (1), the checkpoint is also communicated to the journal which can then move the `logMark` forward and eventually discard old unused journals.
    
    In cae (2), we're not moving the `logMark`. This has few side effects:
     1. When restarting the bookie it will replay a bigger portion of the journal
     2. We are moving the `logMark` only every 1min, so we need to keep 1minute of journals on disk
        - If the disk hosting the journal(s) has low capacity, it can fill it up
        - This is evident when mounting the journal on a 60GB ram-disk and using 8 journals in the bookie. At a sufficient rate, the write cache keeps getting flushed, but the journals are only trimmed after 1 minute, thus accumulating 1min worth of writes in the journal device, filling up the space.
    
    Author: Matteo Merli <mm...@apache.org>
    
    Reviewers: Sijie Guo <si...@apache.org>
    
    This closes #1261 from merlimat/db-storage-journal-trimming
---
 .../org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java     | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
index 3a9c805..4beec22 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
@@ -828,7 +828,9 @@ public class DbLedgerStorage implements CompactableLedgerStorage {
 
     @Override
     public void flush() throws IOException {
-        checkpoint(Checkpoint.MAX);
+        Checkpoint cp = checkpointSource.newCheckpoint();
+        checkpoint(cp);
+        checkpointSource.checkpointComplete(cp, true);
     }
 
     @Override

-- 
To stop receiving notification emails like this one, please contact
mmerli@apache.org.