[GitHub] [pinot] snleee commented on a change in pull request #7827: Use valid bucket end time instead of segment end time for merge/rollup delay metrics

[GitBox <gi...@apache.org>]

snleee commented on a change in pull request #7827:
URL: https://github.com/apache/pinot/pull/7827#discussion_r758668575



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -439,6 +439,18 @@ private boolean validate(TableConfig tableConfig, String taskType) {
     return true;
   }
 
+  /**
+   * Get the max valid merge bucket end time for the segment, LONG.MIN_VALUE if the segment is not ready to merge
+   */
+  private long getMaxValidBucketEndTimeMsForSegment(SegmentZKMetadata segmentZKMetadata, long bucketMs, long bufferMs) {
+    if (segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs > System.currentTimeMillis() - bufferMs) {

Review comment:
       I would separate out this into 2 lines for better readability.
   
   ```
   e.g.
   long bucketEndTimeForSegment = segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs
   if (bucketEndTimeForSegment > System.currentTimeMillis() - bufferMs()) {
     return Long.MIN_VALUE
   }
   ````
   
   
   By the way, why do we need to use `startTimeMs` here? 
   
   Things that we need to brainstorm: do we have a case where 
   `(segmentZKMetadata.getEndTimeMs() / bucketMs + 1)` is different from `segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs`?
   

##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -439,6 +437,23 @@ private boolean validate(TableConfig tableConfig, String taskType) {
     return true;
   }
 
+  /**
+   * Get the max valid merge bucket end time for the segment, LONG.MIN_VALUE if the segment is not ready to merge
+   */
+  private long getMaxValidBucketEndTimeMsForSegment(SegmentZKMetadata segmentZKMetadata, long bucketMs, long bufferMs) {
+    // Make sure the segment is ready to merge (the first bucket <= now - bufferTime)
+    if (segmentZKMetadata.getStartTimeMs() / bucketMs * bucketMs + bucketMs > System.currentTimeMillis() - bufferMs) {
+      return Long.MIN_VALUE;
+    }
+    // The max bucketEndTime is calculated as the min(segment end time, now - bufferTime) rounded to the bucket
+    // boundary. Notice bucketEndTime is exclusive while segment end time is inclusive.
+    // E.g. if bucketTime = 1d, the rounded segment end time of [10/1 00:00, 10/1 23:59] is 10/2 00:00
+    // The rounded segment end time of [10/1 00:00, 10/2 00:00] is 10/3 00:00
+    long maxBucketEndTimeMs = (segmentZKMetadata.getEndTimeMs() / bucketMs + 1) * bucketMs;
+    maxBucketEndTimeMs = Math.min(maxBucketEndTimeMs, (System.currentTimeMillis() - bufferMs) / bucketMs * bucketMs);

Review comment:
       Please store the value for `System.currentTimeMillis()` once and reuse the same value throughout the function.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org