[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8078: Segment retention for table

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on a change in pull request #8078:
URL: https://github.com/apache/pinot/pull/8078#discussion_r795947645



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/SegmentsValidationAndRetentionConfig.java
##########
@@ -29,6 +29,7 @@
 public class SegmentsValidationAndRetentionConfig extends BaseJsonConfig {
   private String _retentionTimeUnit;
   private String _retentionTimeValue;
+  private String _deletedSegmentRetentionInDays;

Review comment:
       Let's make it `_deletedSegmentRetentionPeriod` (e.g. `1d`, `12h`) to be more flexible

##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
##########
@@ -703,7 +703,14 @@ public synchronized PinotResourceManagerResponse deleteSegments(String tableName
       Preconditions.checkArgument(TableNameBuilder.isTableResource(tableNameWithType),
           "Table name: %s is not a valid table name with type suffix", tableNameWithType);
       HelixHelper.removeSegmentsFromIdealState(_helixZkManager, tableNameWithType, segmentNames);
-      _segmentDeletionManager.deleteSegments(tableNameWithType, segmentNames);
+      TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+      String deletedSegmentRetentionString = tableConfig.getValidationConfig().getDeletedSegmentRetentionInDays();
+      if (deletedSegmentRetentionString != null) {
+        _segmentDeletionManager.deleteSegments(tableNameWithType, segmentNames,
+            Integer.parseInt(deletedSegmentRetentionString));

Review comment:
       Add a try catch around this value parsing. Log a warning if the value is invalid, then fall back to the default retention. We don't want to fail the segment deletion by an invalid retention config




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org