[GitHub] [pinot] zhtaoxiang commented on a diff in pull request #9890: enable MergeRollupTask on realtime tables

[GitBox <gi...@apache.org>]

zhtaoxiang commented on code in PR #9890:
URL: https://github.com/apache/pinot/pull/9890#discussion_r1040029118


##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -61,13 +63,26 @@
 /**
  * A {@link PinotTaskGenerator} implementation for generating tasks of type {@link MergeRollupTask}
  *
- * TODO: Add the support for realtime table
+ * Assumptions:
+ *  - When the MergeRollupTask starts the first time, records older than the bufferTimeMs have already been ingested.

Review Comment:
   Good point! 
   
   If I  understand correctly, we are using `maxEndTimeMs - bufferTimeMs`. See the method `getValidBucketEndTimeMsForSegment` for details.
   
   I will update the comment.



##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -428,20 +446,43 @@ public List<PinotTaskConfig> generateTasks(List<TableConfig> tableConfigs) {
     return pinotTaskConfigs;
   }
 
+  @VisibleForTesting
+  static List<SegmentZKMetadata> filterSegmentsBasedOnStatus(TableType tableType, List<SegmentZKMetadata> allSegments) {
+    if (tableType == TableType.REALTIME) {
+      // For realtime table, filter out
+      // 1. in-progress segments
+      // 2. sealed segments with start time later than the earliest start time of all in progress segments

Review Comment:
   maybe `filter out` is confusing here? Let me change it to `don't process`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org