[GitHub] [pinot] npawar commented on a change in pull request #7605: Update watermark if no segments present for a window. Fixes #6988

[GitBox <gi...@apache.org>]

npawar commented on a change in pull request #7605:
URL: https://github.com/apache/pinot/pull/7605#discussion_r754661298



##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
##########
@@ -317,6 +325,13 @@ private long getWatermarkMs(String realtimeTableName, List<SegmentZKMetadata> co
       realtimeToOfflineSegmentsTaskMetadata = new RealtimeToOfflineSegmentsTaskMetadata(realtimeTableName, watermarkMs);
       _clusterInfoAccessor.setRealtimeToOfflineSegmentsTaskMetadata(realtimeToOfflineSegmentsTaskMetadata);
     }
+
+    _realtimeToOfflineSegmentsTaskMetadata = realtimeToOfflineSegmentsTaskMetadata;
+
     return realtimeToOfflineSegmentsTaskMetadata.getWatermarkMs();
   }
+
+  private void setWatermarkMs(long watermarkMs) {
+    _realtimeToOfflineSegmentsTaskMetadata.setWatermarkMs(watermarkMs);

Review comment:
       this will only set it into the RealtimeToOfflineSegmentsTaskMetadata object. It will not get persisted into zk for the next task.

##########
File path: pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
##########
@@ -204,6 +205,13 @@ public String getTaskType() {
       if (segmentNames.isEmpty() || skipGenerate) {
         LOGGER.info("Found no eligible segments for task: {} with window [{} - {}). Skipping task generation", taskType,
             windowStartMs, windowEndMs);
+        if (!skipGenerate) {

Review comment:
       can we go back to line 164 in this iteration itself, if we detect this? So that we dont have to wait for the next scheduling?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org