[GitHub] [pinot] jackjlli commented on a change in pull request #7827: Use valid bucket end time instead of segment end time for merge/rollup delay metrics

[GitBox <gi...@apache.org>]

jackjlli commented on a change in pull request #7827:
URL: https://github.com/apache/pinot/pull/7827#discussion_r756603260



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -252,15 +252,15 @@ public String getTaskType() {
         long bucketEndMs = bucketStartMs + bucketMs;
         // Create delay metrics even if there's no task scheduled, this helps the case that the controller is restarted
         // but the metrics are not available until the controller schedules a valid task
-        long maxEndTimeMs = Long.MIN_VALUE;
+        long maxBucketEndTimeMs = Long.MIN_VALUE;
         for (SegmentZKMetadata preSelectedSegment : preSelectedSegments) {
-          long currentEndTimeMs = preSelectedSegment.getEndTimeMs();
-          // Compute maxEndTimeMs among segments that are valid for merge
-          if (currentEndTimeMs < System.currentTimeMillis() - bufferMs) {
-            maxEndTimeMs = Math.max(maxEndTimeMs, currentEndTimeMs);
+          // Compute maxBucketEndTimeMs among segments that are valid for merge
+          long currentMaxBucketEndTimeMs = getMaxValidBucketEndTimeMsForSegment(preSelectedSegment, bucketMs, bufferMs);
+          if (currentMaxBucketEndTimeMs != Long.MIN_VALUE) {

Review comment:
       This if block may not be needed.

##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -439,6 +439,18 @@ private boolean validate(TableConfig tableConfig, String taskType) {
     return true;
   }
 
+  /**
+   * Get the max valid merge bucket end time for the segment, LONG.MIN_VALUE if the segment is not ready to merge
+   */
+  private long getMaxValidBucketEndTimeMsForSegment(SegmentZKMetadata segmentZKMetadata, long bucketMs, long bufferMs) {
+    if (segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs > System.currentTimeMillis() - bufferMs) {

Review comment:
       Could you add more description on how the max valid bucket end time is calculated?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org