[GitHub] [pinot] kriti-sc commented on pull request #7605: Update watermark if no segments present for a window. Fixes #6988

[GitBox <gi...@apache.org>]

kriti-sc commented on pull request #7605:
URL: https://github.com/apache/pinot/pull/7605#issuecomment-949885810


   @npawar
   If a segment is `CONSUMING` or `DONE`, this check is already being performed [here](https://github.com/apache/pinot/blob/1bd899c9ba45676d1ac25979274391431bdf5ce9/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java#L132), i.e., at the beginning of `RealtimeToOfflineSegment` task generation.
   
   [This check](https://github.com/apache/pinot/blob/1bd899c9ba45676d1ac25979274391431bdf5ce9/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java#L187) also ensures that a `CONSUMING` segment is not processed in `RealtimeToOfflineSegmentTask`.
   
   This PR only progresses the watermark if no segment is found for those segments marked as `DONE` in our processing window.
   
   The solution you are proposing will not work if the segments in the next window are still in `CONSUMING` state at time of progressing the watermark. 
   
   Please let me know your thoughts on all of the above. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org