[jira] [Created] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

["Arun Mahadevan (JIRA)" <ji...@apache.org>]

Arun Mahadevan created SPARK-26655:
--------------------------------------

             Summary: Support multiple aggregates in Structured Streaming append mode
                 Key: SPARK-26655
                 URL: https://issues.apache.org/jira/browse/SPARK-26655
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Arun Mahadevan


Right now multiple aggregates are not supported in structured streaming.

However, in append mode, the aggregates are emitted only after the watermark passes the threshold (e.g. the window boundary) and the emitted value is not affected by further late data. So it possible to chain multiple aggregates in 'Append' output mode without worrying about retractions.

However the current event time watermarks in structured streaming are tracked at a global level and this does not work when aggregates are chained.

We need to track the watermarks at individual operator level so that each operator can make progress independently and not rely on global min or max value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org