You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shuturmurgh (via GitHub)" <gi...@apache.org> on 2026/05/13 09:15:00 UTC

[PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

shuturmurgh opened a new pull request, #18486:
URL: https://github.com/apache/pinot/pull/18486

   Fixes https://github.com/apache/pinot/issues/18485
   
   ## Summary
   
   Add controller capability to reject segment uploads when the segment is already outside the table’s configured **data retention** window (time-based retention), using `RetentionUtils` so the boundary matches retention manager behavior.
   
   - New controller property: `controller.segment.upload.rejectOutOfRetention.enabled` (default `false`). No behavior change until it is set to `true`.
   - Wired on **single-segment upload** only (`PinotSegmentUploadDownloadRestletResource` → `SegmentValidationUtils.rejectUploadIfOutOfRetention`). **METADATA** batch upload and **reingest** upload paths are unchanged.
   - Retention check is **skipped** for **OFFLINE** tables whose batch segment ingestion type is **not** `APPEND` (completed-segment semantics). **REALTIME** and **OFFLINE** with **APPEND** ingestion are evaluated when the flag is on and retention parses successfully.
   - Shared parsing / purgeability helpers live in `RetentionUtils` (`parseTableDataRetentionMillis`, `SegmentMetadata`-based `isPurgeable`, `segmentMetadataEndTimeMillis`). Configuration is surfaced via `ControllerConf`.
   
   ## Release notes
   
   - New controller configuration: `controller.segment.upload.rejectOutOfRetention.enabled` (default `false`). When `true`, single-segment upload may return **403** if the segment is outside the table data retention window as defined by `SegmentsValidationAndRetentionConfig` retention time value/unit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "xiangfu0 (via GitHub) via commits" <co...@pinot.apache.org>.
xiangfu0 commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-6834628046

   Documentation for this change has been added: https://github.com/pinot-contrib/pinot-docs/pull/1178 (merged). It documents the new controller config `controller.segment.upload.reject.out.of.retention` (default `false`) in the controller configuration reference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3239738514


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {
+    return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata), segmentMetadata.getIndexCreationTime(),
+        retentionMs, currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentMetadata.getName());
+  }
+
+  private static boolean isPurgeableInternal(long endTimeMs, long creationTimeMs, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback, String tableNameWithType,
+      String segmentName) {
     if (TimeUtils.timeValueInValidRange(endTimeMs)) {
       return currentTimeMs - endTimeMs > retentionMs;
     }
+    if (useCreationTimeFallback && TimeUtils.timeValueInValidRange(creationTimeMs)) {

Review Comment:
   The creationTime fallback may be of no value for offline tables. It means that if a valid endTime was not computed for data then we fall back to the time when the segment was created. In many cases the createTimeMs will be closer to now(). so `currentTimeMs - creationTimeMs > retentionMs;` will always be false in fallback.
   
   
   In case of upsertCompaction like minion tasks, the creationTime is preserved from the original segments so thay may be very old. The segments would get rejected here. It looks like a similar handling is done previously on minion side to avoid uploading segments near retention boundary: https://github.com/apache/pinot/pull/18285



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {

Review Comment:
   This exception is silently ignored.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {

Review Comment:
   nit: naming does not imply `get` behaviour.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {
+    Interval interval = segmentMetadata.getTimeInterval();
+    if (interval != null && TimeUtils.isValidTimeInterval(interval)) {
+      return interval.getEndMillis();
+    }
+    TimeUnit timeUnit = segmentMetadata.getTimeUnit();

Review Comment:
   why do we need this fallback, generally interval and endtime are set together:
   https://github.com/apache/pinot/blob/21c88f716f376bb7a0238e0feeeff12b088f55f7/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java#L181



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -57,6 +63,56 @@ public static void validateTimeInterval(SegmentMetadata segmentMetadata, TableCo
     }
   }
 
+  /**
+   * When {@code controller.segment.upload.rejectOutOfRetention.enabled} is true on the controller, rejects the upload
+   * if segment end time (or index creation time when {@code controller.retentionManager.enableCreationTimeFallback} is
+   * enabled and end time is invalid) is past the table's {@code retentionTimeValue}/{@code retentionTimeUnit} window,
+   * using the same boundary as the controller retention manager.
+   *
+   * <p>Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+   * invokes this for single-segment upload only. METADATA batch upload ({@code POST /segments/batchUpload}) and
+   * reingested-segment upload do not call it.
+   *
+   * <p>For <b>OFFLINE</b> tables, when batch segment ingestion type is not <b>APPEND</b>, this method returns without
+   * evaluating retention (aligned with time retention for completed segments). <b>REALTIME</b> tables always proceed
+   * to the retention comparison when the flag and parseable retention are set.
+   */
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,

Review Comment:
   Let's add metrics when we reject a segment 



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -57,6 +63,56 @@ public static void validateTimeInterval(SegmentMetadata segmentMetadata, TableCo
     }
   }
 
+  /**
+   * When {@code controller.segment.upload.rejectOutOfRetention.enabled} is true on the controller, rejects the upload
+   * if segment end time (or index creation time when {@code controller.retentionManager.enableCreationTimeFallback} is
+   * enabled and end time is invalid) is past the table's {@code retentionTimeValue}/{@code retentionTimeUnit} window,
+   * using the same boundary as the controller retention manager.
+   *
+   * <p>Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+   * invokes this for single-segment upload only. METADATA batch upload ({@code POST /segments/batchUpload}) and
+   * reingested-segment upload do not call it.
+   *
+   * <p>For <b>OFFLINE</b> tables, when batch segment ingestion type is not <b>APPEND</b>, this method returns without
+   * evaluating retention (aligned with time retention for completed segments). <b>REALTIME</b> tables always proceed
+   * to the retention comparison when the flag and parseable retention are set.
+   */
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      boolean retentionCreationTimeFallbackEnabled) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (tableConfig.getTableType() == TableType.OFFLINE

Review Comment:
   Let's reuse the logic from https://github.com/apache/pinot/blob/21c88f716f376bb7a0238e0feeeff12b088f55f7/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java#L138
   
   We can move to RetentionUtils class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-5382589700

   @Jackie-Jiang I've addressed Copilot comment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "xiangfu0 (via GitHub) via commits" <co...@pinot.apache.org>.
xiangfu0 commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4432196900


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -130,6 +135,51 @@ private static String getPartitionColumn(InstanceAssignmentConfig instanceAssign
     return replicaGroupPartitionConfig != null ? replicaGroupPartitionConfig.getPartitionColumn() : null;
   }
 
+  /// When `controller.segment.upload.rejectOutOfRetention.enabled` is true on the controller, rejects the upload if
+  /// the segment's data end time (from segment file metadata) is past the table's `retentionTimeValue` /
+  /// `retentionTimeUnit` window, using the same boundary as the controller retention manager. **Index creation time is
+  /// not** used as a fallback here: it can reflect upstream segment timestamps (for example upsert compaction) and is a
+  /// poor proxy for the data window at upload time.
+  ///
+  /// Rejection uses HTTP 403 so callers that enable the controller flag can treat the response as a hard failure.
+  /// Clients that do not enable the flag see no behavior change.
+  ///
+  /// Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+  /// invokes this for single-segment upload only. METADATA batch upload (`POST /segments/batchUpload`) and
+  /// reingested-segment upload do not call it.
+  ///
+  /// For tables where the retention manager does not apply time-based retention to completed segments (offline tables
+  /// whose batch ingestion type is not `APPEND`), this method returns without evaluating retention.
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      @Nullable ControllerMetrics controllerMetrics) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (!RetentionUtils.shouldManageTimeBasedDataRetention(tableConfig)) {
+      return;
+    }
+    OptionalLong retentionMsOpt = RetentionUtils.parseTableDataRetentionMillis(validationConfig);
+    if (retentionMsOpt.isEmpty()) {
+      return;
+    }
+    long retentionMs = retentionMsOpt.getAsLong();
+    if (RetentionUtils.isPurgeable(tableConfig.getTableName(), segmentMetadata, retentionMs, currentTimeMs, false)) {

Review Comment:
   This also applies to minion-produced uploads. `BaseMultipleSegmentsConversionExecutor` pushes to `POST /segments` (unless `batchSegmentUpload` is on, which goes to `/segments/batchUpload` and skips this check). For RealtimeToOffline, `postProcess` only advances the watermark after a successful upload, so a 403 here makes the generator retry the same window on every run. That continues until the source realtime segments age out, and indefinitely with the hybrid retention strategy, because the time boundary can't advance.
   
   Could we either exempt task-originated uploads or at least document this interaction on the config key so operators know before enabling it? Today the only signal is the `SEGMENT_ZK_METADATA_CUSTOM_MAP_MODIFIER` header that minion sends, so a cleaner exemption would need a dedicated marker.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -69,4 +109,43 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /// Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+  /// {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+  /// milliseconds (same interpretation as controller time-based retention).
+  ///
+  /// @return millis, or empty if unset or not parseable
+  public static OptionalLong parseTableDataRetentionMillis(

Review Comment:
   This duplicates the unit/value parsing in `TableConfigRetentionUtils.buildRetentionStrategy`, and the two already drifted once (the `Locale.ROOT` fix). Since this now lives in `pinot-common`, could `buildRetentionStrategy` call `parseTableDataRetentionMillis`? Then the upload check and the purge path can't disagree about what the table's retention is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3239739881


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -57,6 +63,56 @@ public static void validateTimeInterval(SegmentMetadata segmentMetadata, TableCo
     }
   }
 
+  /**
+   * When {@code controller.segment.upload.rejectOutOfRetention.enabled} is true on the controller, rejects the upload
+   * if segment end time (or index creation time when {@code controller.retentionManager.enableCreationTimeFallback} is
+   * enabled and end time is invalid) is past the table's {@code retentionTimeValue}/{@code retentionTimeUnit} window,
+   * using the same boundary as the controller retention manager.
+   *
+   * <p>Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+   * invokes this for single-segment upload only. METADATA batch upload ({@code POST /segments/batchUpload}) and
+   * reingested-segment upload do not call it.
+   *
+   * <p>For <b>OFFLINE</b> tables, when batch segment ingestion type is not <b>APPEND</b>, this method returns without
+   * evaluating retention (aligned with time retention for completed segments). <b>REALTIME</b> tables always proceed
+   * to the retention comparison when the flag and parseable retention are set.
+   */
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      boolean retentionCreationTimeFallbackEnabled) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (tableConfig.getTableType() == TableType.OFFLINE
+        && !"APPEND".equalsIgnoreCase(IngestionConfigUtils.getBatchSegmentIngestionType(tableConfig))) {
+      return;
+    }
+    OptionalLong retentionMsOpt = RetentionUtils.parseTableDataRetentionMillis(validationConfig);
+    if (retentionMsOpt.isEmpty()) {
+      if (StringUtils.isNotEmpty(validationConfig.getRetentionTimeUnit())
+          && StringUtils.isNotEmpty(validationConfig.getRetentionTimeValue())) {
+        LOGGER.warn("Invalid retention {} {} for table {}, skipping upload retention check",
+            validationConfig.getRetentionTimeUnit(), validationConfig.getRetentionTimeValue(),
+            tableConfig.getTableName());
+      }
+      return;
+    }
+    long retentionMs = retentionMsOpt.getAsLong();
+    if (RetentionUtils.isPurgeable(tableConfig.getTableName(), segmentMetadata, retentionMs, currentTimeMs,

Review Comment:
   Should we return a no op status code instead of an error. This will require existing clients to do exception handling now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-5133395759

   LGTM, might need to rebase again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-4461587412

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18486?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `83.33333%` with `9 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 34.98%. Comparing base ([`5ebd445`](https://app.codecov.io/gh/apache/pinot/commit/5ebd445336ff295c231b02afa9f202418e296546?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`31f1ba0`](https://app.codecov.io/gh/apache/pinot/commit/31f1ba06318d376d507f0c3f804d09f287b62050?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 32 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18486?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../org/apache/pinot/common/utils/RetentionUtils.java](https://app.codecov.io/gh/apache/pinot/pull/18486?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2FRetentionUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvUmV0ZW50aW9uVXRpbHMuamF2YQ==) | 78.57% | [3 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../controller/api/upload/SegmentValidationUtils.java](https://app.codecov.io/gh/apache/pinot/pull/18486?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fapi%2Fupload%2FSegmentValidationUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvdXBsb2FkL1NlZ21lbnRWYWxpZGF0aW9uVXRpbHMuamF2YQ==) | 83.33% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18486      +/-   ##
   ============================================
   - Coverage     34.99%   34.98%   -0.02%     
   + Complexity      869      863       -6     
   ============================================
     Files          3256     3266      +10     
     Lines        199548   199941     +393     
     Branches      30986    31055      +69     
   ============================================
   + Hits          69840    69955     +115     
   - Misses       123570   123828     +258     
   - Partials       6138     6158      +20     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <83.33%> (-0.02%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <83.33%> (-0.02%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <83.33%> (-0.01%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18486/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <83.33%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18486?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4432677515


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -130,6 +135,51 @@ private static String getPartitionColumn(InstanceAssignmentConfig instanceAssign
     return replicaGroupPartitionConfig != null ? replicaGroupPartitionConfig.getPartitionColumn() : null;
   }
 
+  /// When `controller.segment.upload.rejectOutOfRetention.enabled` is true on the controller, rejects the upload if
+  /// the segment's data end time (from segment file metadata) is past the table's `retentionTimeValue` /
+  /// `retentionTimeUnit` window, using the same boundary as the controller retention manager. **Index creation time is
+  /// not** used as a fallback here: it can reflect upstream segment timestamps (for example upsert compaction) and is a
+  /// poor proxy for the data window at upload time.
+  ///
+  /// Rejection uses HTTP 403 so callers that enable the controller flag can treat the response as a hard failure.
+  /// Clients that do not enable the flag see no behavior change.
+  ///
+  /// Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+  /// invokes this for single-segment upload only. METADATA batch upload (`POST /segments/batchUpload`) and
+  /// reingested-segment upload do not call it.
+  ///
+  /// For tables where the retention manager does not apply time-based retention to completed segments (offline tables
+  /// whose batch ingestion type is not `APPEND`), this method returns without evaluating retention.
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      @Nullable ControllerMetrics controllerMetrics) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (!RetentionUtils.shouldManageTimeBasedDataRetention(tableConfig)) {
+      return;
+    }
+    OptionalLong retentionMsOpt = RetentionUtils.parseTableDataRetentionMillis(validationConfig);
+    if (retentionMsOpt.isEmpty()) {
+      return;
+    }
+    long retentionMs = retentionMsOpt.getAsLong();
+    if (RetentionUtils.isPurgeable(tableConfig.getTableName(), segmentMetadata, retentionMs, currentTimeMs, false)) {

Review Comment:
   Good catch, thanks for tracing the watermark behavior. I've documented this on the config key in `ControllerConf` and in the `rejectUploadIfOutOfRetention` Javadoc (75d3a2c). The docs now say:
   
   - Purge, UpsertCompaction, and RefreshSegment always upload through POST `/segments`, so they are always subject to the check.
   - RealtimeToOffline, MergeRollup, and UpsertCompactMerge are subject to it unless batchSegmentUpload is enabled.
   - A rejected RealtimeToOffline upload doesn't advance the watermark, so the same window is retried until the source segments age out. With the hybrid retention strategy that can go on indefinitely, so operators should check task schedules and buffer periods against table retention before enabling the flag.
   
   I didn't add an exemption in this PR. As you noted, the only signal today is the SEGMENT_ZK_METADATA_CUSTOM_MAP_MODIFIER header, which is too indirect. A clean exemption needs a dedicated marker header from minion, and that change has mixed-version implications. I'd rather do it as a follow-up. Happy to open an issue for it if that works for you.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -130,6 +135,51 @@ private static String getPartitionColumn(InstanceAssignmentConfig instanceAssign
     return replicaGroupPartitionConfig != null ? replicaGroupPartitionConfig.getPartitionColumn() : null;
   }
 
+  /// When `controller.segment.upload.rejectOutOfRetention.enabled` is true on the controller, rejects the upload if
+  /// the segment's data end time (from segment file metadata) is past the table's `retentionTimeValue` /
+  /// `retentionTimeUnit` window, using the same boundary as the controller retention manager. **Index creation time is
+  /// not** used as a fallback here: it can reflect upstream segment timestamps (for example upsert compaction) and is a
+  /// poor proxy for the data window at upload time.
+  ///
+  /// Rejection uses HTTP 403 so callers that enable the controller flag can treat the response as a hard failure.
+  /// Clients that do not enable the flag see no behavior change.
+  ///
+  /// Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+  /// invokes this for single-segment upload only. METADATA batch upload (`POST /segments/batchUpload`) and
+  /// reingested-segment upload do not call it.
+  ///
+  /// For tables where the retention manager does not apply time-based retention to completed segments (offline tables
+  /// whose batch ingestion type is not `APPEND`), this method returns without evaluating retention.
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      @Nullable ControllerMetrics controllerMetrics) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (!RetentionUtils.shouldManageTimeBasedDataRetention(tableConfig)) {
+      return;
+    }
+    OptionalLong retentionMsOpt = RetentionUtils.parseTableDataRetentionMillis(validationConfig);
+    if (retentionMsOpt.isEmpty()) {
+      return;
+    }
+    long retentionMs = retentionMsOpt.getAsLong();
+    if (RetentionUtils.isPurgeable(tableConfig.getTableName(), segmentMetadata, retentionMs, currentTimeMs, false)) {

Review Comment:
   Good catch, thanks for tracing the watermark behavior. I've documented this on the config key in `ControllerConf` and in the `rejectUploadIfOutOfRetention` Javadoc (75d3a2c). The docs now say:
   
   - Purge, UpsertCompaction, and RefreshSegment always upload through POST `/segments`, so they are always subject to the check.
   - RealtimeToOffline, MergeRollup, and UpsertCompactMerge are subject to it unless batchSegmentUpload is enabled.
   - A rejected RealtimeToOffline upload doesn't advance the watermark, so the same window is retried until the source segments age out. With the hybrid retention strategy that can go on indefinitely, so operators should check task schedules and buffer periods against table retention before enabling the flag.
   
   I didn't add an exemption in this PR. As you noted, the only signal today is the `SEGMENT_ZK_METADATA_CUSTOM_MAP_MODIFIER` header, which is too indirect. A clean exemption needs a dedicated marker header from minion, and that change has mixed-version implications. I'd rather do it as a follow-up. Happy to open an issue for it if that works for you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4502314366


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -103,6 +108,59 @@ private static String getPartitionColumn(TableConfig tableConfig) {
     return null;
   }
 
+  /// When `controller.segment.upload.rejectOutOfRetention.enabled` is true on the controller, rejects the upload if
+  /// the segment's data end time (from segment file metadata) is past the table's `retentionTimeValue` /
+  /// `retentionTimeUnit` window. This matches the retention manager's time-based boundary. It does not model the
+  /// hybrid retention strategy (`controller.enable.hybrid.table.retention.strategy`), under which the REALTIME side of
+  /// a hybrid table is purged by the offline time boundary instead of the retention window. The two boundaries are
+  /// independent there, so this check can be either stricter or looser than what the retention manager purges.
+  /// **Index creation time is not** used as a fallback here: it can reflect upstream segment timestamps (for example
+  /// upsert compaction) and is a poor proxy for the data window at upload time.
+  ///
+  /// Rejection uses HTTP 403 so callers that enable the controller flag can treat the response as a hard failure.
+  /// Clients that do not enable the flag see no behavior change.
+  ///
+  /// Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+  /// invokes this for single-segment upload only. METADATA batch upload (`POST /segments/batchUpload`) and
+  /// reingested-segment upload do not call it.
+  ///
+  /// Minion task uploads are not exempt. Purge, UpsertCompaction and RefreshSegment always push through this
+  /// single-segment endpoint. RealtimeToOffline, MergeRollup and UpsertCompactMerge do too unless `batchSegmentUpload`
+  /// is enabled for the task. A rejected RealtimeToOffline upload does not advance the task watermark, so the same
+  /// window is retried on every run until the source segments age out.
+  ///
+  /// For tables where the retention manager does not apply time-based retention to completed segments (offline tables
+  /// whose batch ingestion type is not `APPEND`), this method returns without evaluating retention.
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      @Nullable ControllerMetrics controllerMetrics) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {

Review Comment:
   Done in e167b86. Removed:
   
   - the null check in `rejectUploadIfOutOfRetention`
   - the `@Nullable` parameter and null branch in `parseTableDataRetentionMillis`
   - the `parseTableDataRetentionMillis(null)` assertion
   - the mock-only `testRejectUploadSkippedWhenValidationConfigNull`
   
   Missing and invalid retention on a non-null config are still covered by `testRejectUploadSkippedWhenRetentionNotConfigured` and `testRejectUploadSkippedWhenRetentionInvalid`.



##########
pinot-common/src/test/java/org/apache/pinot/common/utils/RetentionUtilsTest.java:
##########
@@ -47,6 +61,77 @@ private static SegmentZKMetadata makeSegmentWithCreationTime(long endTimeMs, lon
     return segment;
   }
 
+  private static SegmentMetadata mockSegmentMetadataMillis(long endTimeRaw, long indexCreationTimeMs) {
+    SegmentMetadata metadata = Mockito.mock(SegmentMetadata.class);

Review Comment:
   Done in e167b86 for `RetentionUtilsTest`, and for the new assertions in `ControllerConfTest` and `SegmentUploadIntegrationTest`. I left the existing qualified `Assert.` calls in those two files as they were, to keep the diff focused.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "tarun11Mavani (via GitHub)" <gi...@apache.org>.
tarun11Mavani commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3241257100


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {
+    return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata), segmentMetadata.getIndexCreationTime(),
+        retentionMs, currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentMetadata.getName());
+  }
+
+  private static boolean isPurgeableInternal(long endTimeMs, long creationTimeMs, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback, String tableNameWithType,
+      String segmentName) {
     if (TimeUtils.timeValueInValidRange(endTimeMs)) {
       return currentTimeMs - endTimeMs > retentionMs;
     }
+    if (useCreationTimeFallback && TimeUtils.timeValueInValidRange(creationTimeMs)) {

Review Comment:
   makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240735448


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {
+    return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata), segmentMetadata.getIndexCreationTime(),
+        retentionMs, currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentMetadata.getName());
+  }
+
+  private static boolean isPurgeableInternal(long endTimeMs, long creationTimeMs, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback, String tableNameWithType,
+      String segmentName) {
     if (TimeUtils.timeValueInValidRange(endTimeMs)) {
       return currentTimeMs - endTimeMs > retentionMs;
     }
+    if (useCreationTimeFallback && TimeUtils.timeValueInValidRange(creationTimeMs)) {

Review Comment:
   For single-segment upload we now call `RetentionUtils.isPurgeable(..., useCreationTimeFallback=false)`, so we only use the segment’s data end time from file metadata.
   Invalid/missing end time remains fail-open (no reject).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "tarun11Mavani (via GitHub) via commits" <co...@pinot.apache.org>.
tarun11Mavani commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-6768854624

   Looks good to merge. @Jackie-Jiang let me know if there are any concern with merging this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240606041


##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -57,6 +63,56 @@ public static void validateTimeInterval(SegmentMetadata segmentMetadata, TableCo
     }
   }
 
+  /**
+   * When {@code controller.segment.upload.rejectOutOfRetention.enabled} is true on the controller, rejects the upload
+   * if segment end time (or index creation time when {@code controller.retentionManager.enableCreationTimeFallback} is
+   * enabled and end time is invalid) is past the table's {@code retentionTimeValue}/{@code retentionTimeUnit} window,
+   * using the same boundary as the controller retention manager.
+   *
+   * <p>Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+   * invokes this for single-segment upload only. METADATA batch upload ({@code POST /segments/batchUpload}) and
+   * reingested-segment upload do not call it.
+   *
+   * <p>For <b>OFFLINE</b> tables, when batch segment ingestion type is not <b>APPEND</b>, this method returns without
+   * evaluating retention (aligned with time retention for completed segments). <b>REALTIME</b> tables always proceed
+   * to the retention comparison when the flag and parseable retention are set.
+   */
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      boolean retentionCreationTimeFallbackEnabled) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {
+      return;
+    }
+    if (tableConfig.getTableType() == TableType.OFFLINE
+        && !"APPEND".equalsIgnoreCase(IngestionConfigUtils.getBatchSegmentIngestionType(tableConfig))) {
+      return;
+    }
+    OptionalLong retentionMsOpt = RetentionUtils.parseTableDataRetentionMillis(validationConfig);
+    if (retentionMsOpt.isEmpty()) {
+      if (StringUtils.isNotEmpty(validationConfig.getRetentionTimeUnit())
+          && StringUtils.isNotEmpty(validationConfig.getRetentionTimeValue())) {
+        LOGGER.warn("Invalid retention {} {} for table {}, skipping upload retention check",
+            validationConfig.getRetentionTimeUnit(), validationConfig.getRetentionTimeValue(),
+            tableConfig.getTableName());
+      }
+      return;
+    }
+    long retentionMs = retentionMsOpt.getAsLong();
+    if (RetentionUtils.isPurgeable(tableConfig.getTableName(), segmentMetadata, retentionMs, currentTimeMs,

Review Comment:
   Added a metric. This behavior is opt-in so it won't break existing clients. 
   Maybe if other contributors have feedback for such a use case we can add another config for no-action mode.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3239744818


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {
+    return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata), segmentMetadata.getIndexCreationTime(),
+        retentionMs, currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentMetadata.getName());
+  }
+
+  private static boolean isPurgeableInternal(long endTimeMs, long creationTimeMs, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback, String tableNameWithType,
+      String segmentName) {
     if (TimeUtils.timeValueInValidRange(endTimeMs)) {
       return currentTimeMs - endTimeMs > retentionMs;
     }
+    if (useCreationTimeFallback && TimeUtils.timeValueInValidRange(creationTimeMs)) {

Review Comment:
   @tarun11Mavani, can you also take a look from upsertCompaction perspective.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240794292


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {
+    Interval interval = segmentMetadata.getTimeInterval();
+    if (interval != null && TimeUtils.isValidTimeInterval(interval)) {
+      return interval.getEndMillis();
+    }
+    TimeUnit timeUnit = segmentMetadata.getTimeUnit();

Review Comment:
   `SegmentMetadataImpl` can fail to build `_timeInterval` while still having raw start/end/unit in some failure paths (`setTimeInfo` catches Exception and logs).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-5251045341

   @shuturmurgh, please specify the necessary labels.
   
   cc: @Jackie-Jiang, @KKcorps could you help with review or assigning reviewers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240794292


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {
+    Interval interval = segmentMetadata.getTimeInterval();
+    if (interval != null && TimeUtils.isValidTimeInterval(interval)) {
+      return interval.getEndMillis();
+    }
+    TimeUnit timeUnit = segmentMetadata.getTimeUnit();

Review Comment:
   - Removed fallback on `timeInterval`



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {
+    Interval interval = segmentMetadata.getTimeInterval();
+    if (interval != null && TimeUtils.isValidTimeInterval(interval)) {
+      return interval.getEndMillis();
+    }
+    TimeUnit timeUnit = segmentMetadata.getTimeUnit();

Review Comment:
   Removed fallback on `timeInterval`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-6544891071

   @Jackie-Jiang can you please help here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on PR #18486:
URL: https://github.com/apache/pinot/pull/18486#issuecomment-6767774034

   @Jackie-Jiang is this good to merge now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4502311509


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -49,18 +69,38 @@ private RetentionUtils() {
   /// @return true if the segment is past the retention boundary, false otherwise
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /// Whether segment file metadata is past the retention window, using end time from
+  /// {@link #getSegmentMetadataEndTimeMillis(SegmentMetadata)} and
+  /// {@link SegmentMetadata#getIndexCreationTime()} as the optional fallback clock. Invalid-end paths emit the same
+  /// style of debug/warn lines as the ZK overload, using
+  /// {@link SegmentMetadata#getName()} for the segment in log messages.
+  ///
+  /// - If end time is valid: purgeable when `currentTimeMs - endTimeMs > retentionMs`.
+  /// - If end time is invalid and `useCreationTimeFallback` is true and index creation time is valid: purgeable when
+  ///   `currentTimeMs - creationTimeMs > retentionMs`.
+  /// - Otherwise: not purgeable (fail-open).
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {

Review Comment:
   Done in e167b86. The overload is now `isPurgeable(tableNameWithType, segmentMetadata, retentionMs, currentTimeMs)`. It reads only the end time and time unit and never calls `getIndexCreationTime()`.
   
   I also restored the `SegmentZKMetadata` overload to its original body, so the creation-time fallback used by the retention manager and the minion task generators is unchanged. The fallback test for the new overload is removed.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "tarun11Mavani (via GitHub)" <gi...@apache.org>.
tarun11Mavani commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240843146


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +77,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #getSegmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */

Review Comment:
   Use md style java doc for new code push.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4432679870


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -69,4 +109,43 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /// Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+  /// {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+  /// milliseconds (same interpretation as controller time-based retention).
+  ///
+  /// @return millis, or empty if unset or not parseable
+  public static OptionalLong parseTableDataRetentionMillis(

Review Comment:
   Agreed. `TableConfigRetentionUtils.buildRetentionStrategy` now delegates to `RetentionUtils.parseTableDataRetentionMillis`, so the upload check and the purge path share one parser (75d3a2c). The resulting boundary is the same as before, and the existing `RetentionManagerTest` cases pass unchanged. I also added `TableConfigRetentionUtilsTest`. It covers the configured window, lowercase units, invalid or missing retention, and propagation of the creation-time fallback flag.
   
   While doing this I also dropped the helper's parse-failure log to DEBUG, so a misconfigured table logs one warning per retention cycle instead of two.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4502307259


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -370,6 +370,17 @@ public static long getRandomInitialDelayInSeconds() {
   // Amount of the time the segment can take from the beginning of upload to the end of upload. Used when parallel push
   // protection is enabled. If the upload does not finish within the timeout, next upload can override the previous one.
   public static final String SEGMENT_UPLOAD_TIMEOUT_IN_MILLIS = "controller.segment.upload.timeoutInMillis";
+  /// If true, the controller may reject single-segment uploads with HTTP 403 when segment data end is past the table
+  /// data retention window (see {@link org.apache.pinot.controller.api.upload.SegmentValidationUtils}).
+  ///
+  /// This also applies to minion task uploads that push through `POST /segments`: always for Purge, UpsertCompaction
+  /// and RefreshSegment, and for RealtimeToOffline, MergeRollup and UpsertCompactMerge unless `batchSegmentUpload` is
+  /// on.
+  /// A rejected RealtimeToOffline upload does not advance its watermark, so the task retries the same window until the
+  /// source segments age out (indefinitely with the hybrid retention strategy, since the time boundary cannot advance).
+  /// Review task schedules and buffer periods against table retention before enabling.
+  public static final String SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION_ENABLED =
+      "controller.segment.upload.rejectOutOfRetention.enabled";

Review Comment:
   Done in e167b86. The key is now `controller.segment.upload.reject.out.of.retention`, still defaulting to `false`.
   
   - Constants: `SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION` and `DEFAULT_SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION`
   - Accessors: `isSegmentUploadRejectOutOfRetention()` and `setSegmentUploadRejectOutOfRetention(boolean)`
   - Also updated: the `SegmentValidationUtils` Javadoc, `SegmentUploadIntegrationTest` and the PR description
   
   No compatibility alias, since the key hasn't been released.



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/TableConfigRetentionUtilsTest.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.retention;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.controller.helix.core.retention.strategy.RetentionStrategy;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+public class TableConfigRetentionUtilsTest {

Review Comment:
   Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "tarun11Mavani (via GitHub) via commits" <co...@pinot.apache.org>.
tarun11Mavani merged PR #18486:
URL: https://github.com/apache/pinot/pull/18486


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240735448


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -54,18 +61,40 @@ private RetentionUtils() {
    */
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /**
+   * Whether segment file metadata is past the retention window, using end time from
+   * {@link #segmentMetadataEndTimeMillis(SegmentMetadata)} and {@link SegmentMetadata#getIndexCreationTime()} as the
+   * optional fallback clock. Invalid-end paths emit the same style of debug/warn lines as the ZK overload, using
+   * {@link SegmentMetadata#getName()} for the segment in log messages.
+   * <ul>
+   *   <li>If end time is valid: purgeable when {@code currentTimeMs - endTimeMs > retentionMs}.</li>
+   *   <li>If end time is invalid and {@code useCreationTimeFallback} is true and index creation time is valid:
+   *       purgeable when {@code currentTimeMs - creationTimeMs > retentionMs}.</li>
+   *   <li>Otherwise: not purgeable (fail-open).</li>
+   * </ul>
+   */
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {
+    return isPurgeableInternal(segmentMetadataEndTimeMillis(segmentMetadata), segmentMetadata.getIndexCreationTime(),
+        retentionMs, currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentMetadata.getName());
+  }
+
+  private static boolean isPurgeableInternal(long endTimeMs, long creationTimeMs, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback, String tableNameWithType,
+      String segmentName) {
     if (TimeUtils.timeValueInValidRange(endTimeMs)) {
       return currentTimeMs - endTimeMs > retentionMs;
     }
+    if (useCreationTimeFallback && TimeUtils.timeValueInValidRange(creationTimeMs)) {

Review Comment:
   For single-segment upload we now call RetentionUtils.isPurgeable(..., useCreationTimeFallback=false), so we only use the segment’s data end time from file metadata.
   Invalid/missing end time remains fail-open (no reject).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub)" <gi...@apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r3240794292


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -74,4 +103,46 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /**
+   * Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+   * {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+   * milliseconds (same interpretation as controller time-based retention).
+   *
+   * @return millis, or empty if unset or not parseable
+   */
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase())
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      return OptionalLong.empty();
+    }
+  }
+
+  /**
+   * End timestamp in epoch millis for retention comparison from segment file metadata, aligned with segment impl time
+   * interval when valid.
+   */
+  static long segmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {
+    Interval interval = segmentMetadata.getTimeInterval();
+    if (interval != null && TimeUtils.isValidTimeInterval(interval)) {
+      return interval.getEndMillis();
+    }
+    TimeUnit timeUnit = segmentMetadata.getTimeUnit();

Review Comment:
   - `SegmentMetadataImpl` can fail to build `_timeInterval` while still having raw start/end/unit in some failure paths (`setTimeInfo` catches Exception and logs).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "Jackie-Jiang (via GitHub) via commits" <co...@pinot.apache.org>.
Jackie-Jiang commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4499111668


##########
pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java:
##########
@@ -370,6 +370,17 @@ public static long getRandomInitialDelayInSeconds() {
   // Amount of the time the segment can take from the beginning of upload to the end of upload. Used when parallel push
   // protection is enabled. If the upload does not finish within the timeout, next upload can override the previous one.
   public static final String SEGMENT_UPLOAD_TIMEOUT_IN_MILLIS = "controller.segment.upload.timeoutInMillis";
+  /// If true, the controller may reject single-segment uploads with HTTP 403 when segment data end is past the table
+  /// data retention window (see {@link org.apache.pinot.controller.api.upload.SegmentValidationUtils}).
+  ///
+  /// This also applies to minion task uploads that push through `POST /segments`: always for Purge, UpsertCompaction
+  /// and RefreshSegment, and for RealtimeToOffline, MergeRollup and UpsertCompactMerge unless `batchSegmentUpload` is
+  /// on.
+  /// A rejected RealtimeToOffline upload does not advance its watermark, so the task retries the same window until the
+  /// source segments age out (indefinitely with the hybrid retention strategy, since the time boundary cannot advance).
+  /// Review task schedules and buffer periods against table retention before enabling.
+  public static final String SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION_ENABLED =
+      "controller.segment.upload.rejectOutOfRetention.enabled";

Review Comment:
   Suggestion: rename this to `controller.segment.upload.reject.out.of.retention`, retaining the `false` default. `reject` already expresses the boolean action, so `.enabled` is redundant. `ControllerConf` mixes styles, but lowercase, dot-separated words fit keys such as `controller.enable.storage.quota.check`. Rename the constants to `SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION` and `DEFAULT_SEGMENT_UPLOAD_REJECT_OUT_OF_RETENTION`, and update the accessors, references, and documentation accordingly. Since this key is new and unreleased, a direct rename does not need a compatibility alias.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -69,4 +109,43 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /// Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+  /// {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+  /// milliseconds (same interpretation as controller time-based retention).
+  ///
+  /// @return millis, or empty if unset or not parseable
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase(Locale.ROOT))
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      LOGGER.debug("Failed to parse table data retention: unit='{}' value='{}'", retentionTimeUnit, retentionTimeValue,
+          e);
+      return OptionalLong.empty();
+    }
+  }
+
+  /// Returns segment end time in epoch millis from on-disk {@link SegmentMetadata} for retention comparison, using
+  /// {@link SegmentMetadata#getTimeUnit()} and {@link SegmentMetadata#getEndTime()} (same raw fields minion task
+  /// executors pass when configuring generation from segment files). When `timeUnit` is non-null and raw end time is
+  /// not {@link Long#MIN_VALUE}, returns `timeUnit.toMillis(endTime)`; otherwise returns the raw end time (typically
+  /// `Long.MIN_VALUE` when unset).
+  public static long getSegmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {

Review Comment:
   Suggestion: make `getSegmentMetadataEndTimeMillis()` private. Outside tests, its only caller is within `RetentionUtils`, so it does not need to become public API. Cover conversion through retention outcomes and remove tests that assert which metadata getters are called; that preserves behavioral coverage without locking down the implementation.



##########
pinot-common/src/test/java/org/apache/pinot/common/utils/RetentionUtilsTest.java:
##########
@@ -47,6 +61,77 @@ private static SegmentZKMetadata makeSegmentWithCreationTime(long endTimeMs, lon
     return segment;
   }
 
+  private static SegmentMetadata mockSegmentMetadataMillis(long endTimeRaw, long indexCreationTimeMs) {
+    SegmentMetadata metadata = Mockito.mock(SegmentMetadata.class);

Review Comment:
   Nit: use static imports for the new Mockito and Assert calls, as required by the repository conventions. The same cleanup applies to the new qualified assertion calls in `ControllerConfTest` and `SegmentUploadIntegrationTest`.



##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/SegmentValidationUtils.java:
##########
@@ -103,6 +108,59 @@ private static String getPartitionColumn(TableConfig tableConfig) {
     return null;
   }
 
+  /// When `controller.segment.upload.rejectOutOfRetention.enabled` is true on the controller, rejects the upload if
+  /// the segment's data end time (from segment file metadata) is past the table's `retentionTimeValue` /
+  /// `retentionTimeUnit` window. This matches the retention manager's time-based boundary. It does not model the
+  /// hybrid retention strategy (`controller.enable.hybrid.table.retention.strategy`), under which the REALTIME side of
+  /// a hybrid table is purged by the offline time boundary instead of the retention window. The two boundaries are
+  /// independent there, so this check can be either stricter or looser than what the retention manager purges.
+  /// **Index creation time is not** used as a fallback here: it can reflect upstream segment timestamps (for example
+  /// upsert compaction) and is a poor proxy for the data window at upload time.
+  ///
+  /// Rejection uses HTTP 403 so callers that enable the controller flag can treat the response as a hard failure.
+  /// Clients that do not enable the flag see no behavior change.
+  ///
+  /// Controller wiring: {@link org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource}
+  /// invokes this for single-segment upload only. METADATA batch upload (`POST /segments/batchUpload`) and
+  /// reingested-segment upload do not call it.
+  ///
+  /// Minion task uploads are not exempt. Purge, UpsertCompaction and RefreshSegment always push through this
+  /// single-segment endpoint. RealtimeToOffline, MergeRollup and UpsertCompactMerge do too unless `batchSegmentUpload`
+  /// is enabled for the task. A rejected RealtimeToOffline upload does not advance the task watermark, so the same
+  /// window is retried on every run until the source segments age out.
+  ///
+  /// For tables where the retention manager does not apply time-based retention to completed segments (offline tables
+  /// whose batch ingestion type is not `APPEND`), this method returns without evaluating retention.
+  public static void rejectUploadIfOutOfRetention(SegmentMetadata segmentMetadata, TableConfig tableConfig,
+      long currentTimeMs, boolean controllerRejectOutOfRetentionEnabled,
+      @Nullable ControllerMetrics controllerMetrics) {
+    if (!controllerRejectOutOfRetentionEnabled) {
+      return;
+    }
+    SegmentsValidationAndRetentionConfig validationConfig = tableConfig.getValidationConfig();
+    if (validationConfig == null) {

Review Comment:
   Suggestion: rely on `TableConfig`'s non-null validation-config invariant here. Both its constructor and `setValidationConfig()` reject null, and all production callers of the new parser obtain the config from `TableConfig`. This guard, the parser's `@Nullable`/null handling, and the mock-only null-config tests can be removed. Keep coverage for a non-null config with missing or invalid retention fields.



##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -49,18 +69,38 @@ private RetentionUtils() {
   /// @return true if the segment is past the retention boundary, false otherwise
   public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata segmentZKMetadata, long retentionMs,
       long currentTimeMs, boolean useCreationTimeFallback) {
-    String segmentName = segmentZKMetadata.getSegmentName();
-    long endTimeMs = segmentZKMetadata.getEndTimeMs();
+    return isPurgeableInternal(segmentZKMetadata.getEndTimeMs(), segmentZKMetadata.getCreationTime(), retentionMs,
+        currentTimeMs, useCreationTimeFallback, tableNameWithType, segmentZKMetadata.getSegmentName());
+  }
+
+  /// Whether segment file metadata is past the retention window, using end time from
+  /// {@link #getSegmentMetadataEndTimeMillis(SegmentMetadata)} and
+  /// {@link SegmentMetadata#getIndexCreationTime()} as the optional fallback clock. Invalid-end paths emit the same
+  /// style of debug/warn lines as the ZK overload, using
+  /// {@link SegmentMetadata#getName()} for the segment in log messages.
+  ///
+  /// - If end time is valid: purgeable when `currentTimeMs - endTimeMs > retentionMs`.
+  /// - If end time is invalid and `useCreationTimeFallback` is true and index creation time is valid: purgeable when
+  ///   `currentTimeMs - creationTimeMs > retentionMs`.
+  /// - Otherwise: not purgeable (fail-open).
+  public static boolean isPurgeable(String tableNameWithType, SegmentMetadata segmentMetadata, long retentionMs,
+      long currentTimeMs, boolean useCreationTimeFallback) {

Review Comment:
   Suggestion: remove `useCreationTimeFallback` from this new `SegmentMetadata` overload and avoid reading `getIndexCreationTime()`. Its only production caller passes `false`, and the upload contract explicitly excludes index-creation-time fallback. Keeping the option adds API and tests for behavior this feature does not use. Preserve creation-time fallback in the existing `SegmentZKMetadata` overload.



##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/retention/TableConfigRetentionUtilsTest.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.retention;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.controller.helix.core.retention.strategy.RetentionStrategy;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+public class TableConfigRetentionUtilsTest {

Review Comment:
   Nit: add class-level Javadoc describing the retention-strategy behavior covered by this new test class, per the repository convention for new classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add controller support for rejecting out-of-retention segment uploads [pinot]

Posted by "shuturmurgh (via GitHub) via commits" <co...@pinot.apache.org>.
shuturmurgh commented on code in PR #18486:
URL: https://github.com/apache/pinot/pull/18486#discussion_r4502312887


##########
pinot-common/src/main/java/org/apache/pinot/common/utils/RetentionUtils.java:
##########
@@ -69,4 +109,43 @@ public static boolean isPurgeable(String tableNameWithType, SegmentZKMetadata se
     }
     return false;
   }
+
+  /// Parses {@link SegmentsValidationAndRetentionConfig#getRetentionTimeUnit()} and
+  /// {@link SegmentsValidationAndRetentionConfig#getRetentionTimeValue()} into table data retention duration in
+  /// milliseconds (same interpretation as controller time-based retention).
+  ///
+  /// @return millis, or empty if unset or not parseable
+  public static OptionalLong parseTableDataRetentionMillis(
+      @Nullable SegmentsValidationAndRetentionConfig validationConfig) {
+    if (validationConfig == null) {
+      return OptionalLong.empty();
+    }
+    String retentionTimeUnit = validationConfig.getRetentionTimeUnit();
+    String retentionTimeValue = validationConfig.getRetentionTimeValue();
+    if (StringUtils.isEmpty(retentionTimeUnit) || StringUtils.isEmpty(retentionTimeValue)) {
+      return OptionalLong.empty();
+    }
+    try {
+      return OptionalLong.of(TimeUnit.valueOf(retentionTimeUnit.toUpperCase(Locale.ROOT))
+          .toMillis(Long.parseLong(retentionTimeValue)));
+    } catch (Exception e) {
+      LOGGER.debug("Failed to parse table data retention: unit='{}' value='{}'", retentionTimeUnit, retentionTimeValue,
+          e);
+      return OptionalLong.empty();
+    }
+  }
+
+  /// Returns segment end time in epoch millis from on-disk {@link SegmentMetadata} for retention comparison, using
+  /// {@link SegmentMetadata#getTimeUnit()} and {@link SegmentMetadata#getEndTime()} (same raw fields minion task
+  /// executors pass when configuring generation from segment files). When `timeUnit` is non-null and raw end time is
+  /// not {@link Long#MIN_VALUE}, returns `timeUnit.toMillis(endTime)`; otherwise returns the raw end time (typically
+  /// `Long.MIN_VALUE` when unset).
+  public static long getSegmentMetadataEndTimeMillis(SegmentMetadata segmentMetadata) {

Review Comment:
   Done in e167b86. I removed the getter tests, including the ones that verified `getTimeInterval()` is never called. The conversion is now covered through `isPurgeable` outcomes:
   
   - millisecond end time at the boundary and 1 ms past it
   - `DAYS` end time converted around the 7-day window
   - invalid end time
   - no time metadata (null unit, `Long.MIN_VALUE` end time)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org