[GitHub] [spark] viirya commented on a change in pull request #31570: [WIP][SPARK-10816][SS] SessionWindow support for Structure Streaming

[GitBox <gi...@apache.org>]

viirya commented on a change in pull request #31570:
URL: https://github.com/apache/spark/pull/31570#discussion_r576596507



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala
##########
@@ -262,6 +262,17 @@ object AggUtils {
    *  - PartialMerge (now there is at most 1 tuple per group)
    *  - StateStoreSave (saves the tuple for the next batch)
    *  - Complete (output the current result of the aggregation)
+   *
+   * Plans a streaming aggregation with Session Window using the following progression:
+   *  - (Shuffle + Session Window Assignment, see `SessionWindowExec`)
+   *  - Partial Aggregation (now there is at most 1 tuple per group)
+   *  - SessionStateStoreRestore (now there is 1 tuple from this batch + optionally one from

Review comment:
       This process was simplified, compared with original PR. Removed unnecessary `PartialMerge`s so the session window streaming aggregation has the same physical structure as general streaming aggregation.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org