[GitHub] merlimat commented on issue #1949: [bookie-gc] add option for persistent entry-log metadata map

[GitBox <gi...@apache.org>]

merlimat commented on issue #1949: [bookie-gc] add option for persistent entry-log metadata map
URL: https://github.com/apache/bookkeeper/pull/1949#issuecomment-464419250
 
 
   > @sijie @merlimat do we seriously have an issue where we need to handle large number of entrylogs ? This all came out of ZK/GC issue and it will be taken care.
   
   I haven't checked this PR yet (will do soon :) ). 
   
   The main issue is that since the entryLogMaps for each entryLog are kept in memory. The amount of memory needed by a bookie is directly proportional to the amount of data stored. Your milage might vary: if you have all your data in 10 ledgers this will require less memory that if you have it in 10M ledgers, because for each entryLog we track (ledgerId -> sizeOfEntriesInThisLog).. 
   
   In any case, I believe this is a fundamental issue with the current approach, in that we need to be able to have a fixed amount of memory used, independent of the amount of data.
   
   In many cases, with bookies holding >10TB we saw this very high mem usage, leading to issue where the heap size is not enough and bookie basically restart and goes into a loop of GC leading to OOM.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services