[GitHub] [pinot] jtao15 commented on a diff in pull request #9890: enable MergeRollupTask on realtime tables

[GitBox <gi...@apache.org>]

jtao15 commented on code in PR #9890:
URL: https://github.com/apache/pinot/pull/9890#discussion_r1039936458


##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -61,13 +63,26 @@
 /**
  * A {@link PinotTaskGenerator} implementation for generating tasks of type {@link MergeRollupTask}
  *
- * TODO: Add the support for realtime table
+ * Assumptions:
+ *  - When the MergeRollupTask starts the first time, records older than the bufferTimeMs have already been ingested.

Review Comment:
   Currently, we make sure `bucketEndMs <= now - bufferTimeMs` to process the time bucket. What if we use maxEndTimeMs of valid segments instead of now? Will it help the bootstrap and pause/resume case?



##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java:
##########
@@ -428,20 +446,43 @@ public List<PinotTaskConfig> generateTasks(List<TableConfig> tableConfigs) {
     return pinotTaskConfigs;
   }
 
+  @VisibleForTesting
+  static List<SegmentZKMetadata> filterSegmentsBasedOnStatus(TableType tableType, List<SegmentZKMetadata> allSegments) {
+    if (tableType == TableType.REALTIME) {
+      // For realtime table, filter out
+      // 1. in-progress segments
+      // 2. sealed segments with start time later than the earliest start time of all in progress segments

Review Comment:
   (nit) sealed segments with start time `earlier` than?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org