[GitHub] [pinot] jtao15 commented on a change in pull request #7424: Fix retention for cleaning up segment lineage

[GitBox <gi...@apache.org>]

jtao15 commented on a change in pull request #7424:
URL: https://github.com/apache/pinot/pull/7424#discussion_r707593664



##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
##########
@@ -246,8 +251,12 @@ private void manageSegmentLineageCleanupForTable(String tableNameWithType) {
             .writeSegmentLineage(_pinotHelixResourceManager.getPropertyStore(), segmentLineage, expectedVersion)) {
           // Delete segments based on the segment lineage
           _pinotHelixResourceManager.deleteSegments(tableNameWithType, segmentsToDelete);
+          LOGGER.info("Finished cleaning up segment lineage for table: {} in {}ms", tableNameWithType,
+              (System.currentTimeMillis() - cleanupStartTime));
           return true;
         } else {
+          LOGGER.info("Failed to write segment lineage back when cleaning up segment lineage for table: {}",

Review comment:
       This could be a `warn`? With current logic, this usually should not happen. This can only happen when `startSegmentReplace` or `endSegmentReplace` is called at the same time when retention process is running?  

##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
##########
@@ -80,7 +80,13 @@ public RetentionManager(PinotHelixResourceManager pinotHelixResourceManager,
   @Override
   protected void processTable(String tableNameWithType) {
     LOGGER.info("Start managing retention for table: {}", tableNameWithType);
+    // Manage normal table retention except segment lineage cleanup.
+    // The reason of separating the logic is that REFRESH only table will be skipped in the first part,
+    // whereas the segment lineage cleanup needs to be handled.
     manageRetentionForTable(tableNameWithType);

Review comment:
       Maybe update line 111 as well to indicate it skip managing retention only?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org