[GitHub] [bookkeeper] hangc0276 commented on a diff in pull request #3631: Add small files check in garbage collection

["hangc0276 (via GitHub)" <gi...@apache.org>]

hangc0276 commented on code in PR #3631:
URL: https://github.com/apache/bookkeeper/pull/3631#discussion_r1133574229


##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -548,14 +548,24 @@ private boolean removeIfLedgerNotExists(EntryLogMetadata meta) throws EntryLogMe
     @VisibleForTesting
     void doCompactEntryLogs(double threshold, long maxTimeMillis) throws EntryLogMetadataMapException {
         LOG.info("Do compaction to compact those files lower than {}", threshold);
+        double tmpGcEntryLogSizeRatio = conf.getGcEntryLogSizeRatio() <= 0 ? 0 : conf.getGcEntryLogSizeRatio();
+        final double gcEntryLogSizeRatio = tmpGcEntryLogSizeRatio > 1 ? 1.0 : tmpGcEntryLogSizeRatio;
+        if (gcEntryLogSizeRatio > 0.5) {
+            LOG.warn("Configured gcEntryLogSizeRatio: {}, updated gcEntryLogSizeRatio: {} gratter than 0.5, "
+                + "which means any entryLog file size less than {} will be compacted and it will bring heavy pressure "
+                + "on garbage collection.", conf.getGcEntryLogSizeRatio(), gcEntryLogSizeRatio,
+                gcEntryLogSizeRatio * conf.getEntryLogSizeLimit());
+        }
 
         final int numBuckets = 10;
         int[] entryLogUsageBuckets = new int[numBuckets];
         int[] compactedBuckets = new int[numBuckets];
 
         ArrayList<LinkedList<Long>> compactableBuckets = new ArrayList<>(numBuckets);
+        ArrayList<LinkedList<Long>> smallFilesCompactableBuckets = new ArrayList<>(numBuckets);

Review Comment:
   @dlg99 Good suggestion. I updated the code with `useTargetEntryLogSizeForGc` flag.  Please help take a look, thanks.
   
   > As an improvement, transactional compaction can fallback to a regular one if the estimated size of entry log after compaction is below some limit.
   
   For this one, I will open a new PR to handle it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org