You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2020/11/22 19:36:24 UTC

[GitHub] [bookkeeper] MarvinCai opened a new pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

MarvinCai opened a new pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494


   Descriptions of the changes in this PR:
   
   
   
   ### Motivation
   
   Try add read ahead capability to ledgerhandler.
   
   ### Changes
   
   Try to implement read ahead logic in ledgerhandler similar to what DLog.
   Try to prefetch entry in order from dirstentry(0) and repeat till cache is full.
   Try to fulfill read request from cache first and file read request if entry is not in cache and combine the final result.
   
   Master Issue: #apache/pulsar#3804
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on a change in pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#discussion_r531485307



##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
##########
@@ -173,6 +184,33 @@
     final Counter lacUpdateMissesCounter;
     private final OpStatsLogger clientChannelWriteWaitStats;
 
+    /**
+     * Read ahead related parameters
+     */
+    // Number of read ahead entries allowed in cache.
+    private final int maxNumReadAheadEntries = 0;

Review comment:
       can we group all of these fields into a separate datastructure ?
   all of these fields are useful only in case of read ahead feature enabled




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] MarvinCai commented on pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#issuecomment-778835259


   close this one in flavor of : https://github.com/apache/pulsar/pull/9056


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] MarvinCai commented on pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#issuecomment-734168859


   @eolivelli hi, no worry
   I'm trying to add the readahead by using a map of [entryId -> content] as readahead entry cache of size X
   Entry will be read into the cache by the doReadAhead method which will read Y entry a time and keep reading till fill the cache or encounter error.
   The cached entry will have ttl and will be evicted when expired, a background task should also try to fill the cache at the same rate of entry expiration (instead of delete cached entry every time it's used to fulfill a read request cause I thought it's possible an entry might be read multiple times)
   When read request come in we'll first try to search by entryId in cache to fulfill the request, if it missed we'll try to read with ledgerHandler and do concatenation if necessary (in case some of read operation for cached entry failed) 
   I'll definitely have test case for it but just want to make sure this sounds like a valid approach


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] MarvinCai commented on pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#issuecomment-731832276


   @eolivelli I'm trying to add read ahead in ledgerhandler, this is like initial implementation, can you help take a look to see if I'm on the right track or if I'm missing something. Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] MarvinCai closed pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
MarvinCai closed pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] eolivelli commented on pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#issuecomment-733816581


   @MarvinCai I missed your message, sorry for late replay.
   
   Can you please tell more about your approach ?
   
   It would be also good to start to add tests in order to demonstrate the feature.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [bookkeeper] MarvinCai commented on pull request #2494: [WIP][LedgerHandler] Try add read ahead capability to ledgerhandler.

Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #2494:
URL: https://github.com/apache/bookkeeper/pull/2494#issuecomment-731831454


   Currently using a map with expiration as cache, but it can't release entry when evicting expired entries. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org