[GitHub] rdhabalia commented on a change in pull request #1938: [bk-gc] Prevent OOM: Restrict number of entry-loggers extraction

[GitBox <gi...@apache.org>]

rdhabalia commented on a change in pull request #1938: [bk-gc] Prevent OOM: Restrict number of entry-loggers extraction
URL: https://github.com/apache/bookkeeper/pull/1938#discussion_r257442532
 
 

 ##########
 File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
 ##########
 @@ -322,55 +332,64 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
         }
         // Recover and clean up previous state if using transactional compaction
         compactor.cleanUpAndRecover();
+        
+        long startTime = System.currentTimeMillis();
+        boolean isIteration = false;
+        do {
+            // Extract all of the ledger ID's that comprise all of the entry
+            // logs
+            // (except for the current new one which is still being written to).
+            entryLogMetaMap = extractMetaFromEntryLogs(entryLogMetaMap, isIteration);
+
+            // gc inactive/deleted ledgers
+            doGcLedgers();
 
 Review comment:
   yes, that's correct. therefore, have provided option `maxEntryLoggersPerScan=2000`. In normal case, bookie will have < 500 entryLog files so, it will not require multiple iteration but it will target issue where bookie is not able to recover with large number of entrylog files. and therefore, @merlimat suggested to persist entryLogmetadataMap into rocksDB so, I have created PR: #1949 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services