[jira] [Created] (SAMZA-2801) Support excluding tasks from watermark computation when exceeding idle time

["Xinyu Liu (Jira)" <ji...@apache.org>]

Xinyu Liu created SAMZA-2801:
--------------------------------

             Summary: Support excluding tasks from watermark computation when exceeding idle time
                 Key: SAMZA-2801
                 URL: https://issues.apache.org/jira/browse/SAMZA-2801
             Project: Samza
          Issue Type: Improvement
            Reporter: Xinyu Liu
            Assignee: Xinyu Liu


Currently in the Samza event-time watermark aggregation logic, it will compute the watermark as the min of watermarks from all upstream tasks. However, in the lagging cases, the upstream task might not generate watermark for a long period. In this case, the new watermarks will not be generated and the downstream aggregation will be stuck.

To address this issue, we will implement an mechanism to exclude the tasks that have been "idle" in generating watermark for a configured time, so that the aggregated watermarks will still be generated. Note this mechanism will unblock downstream, but also at the risk of moving eventtime clock faster and the events from lagging tasks will become late arrivals.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)