[GitHub] [spark] HeartSaVioR commented on a change in pull request #35762: [SPARK-38443][SS][DOC] Document config STREAMING_SESSION_WINDOW_MERGE_SESSIONS_IN_LOCAL_PARTITION

[GitBox <gi...@apache.org>]

HeartSaVioR commented on a change in pull request #35762:
URL: https://github.com/apache/spark/pull/35762#discussion_r822169436



##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1225,6 +1225,12 @@ Note that there are some restrictions when you use session window in streaming q
 
 For batch query, global window (only having `session_window` in grouping key) is supported.
 
+Due to implementation details of session window in streaming query, for each input row in same session,

Review comment:
       Why not explain the behavior of default and which case it helps? If we don't explain the case default is better, we fail to let end users reason about why this config exists, as well as why we disable this by default.
   
   How about revising the content like this?
   
   ````
   By default, Spark does not perform partial aggregation for session window aggregation, since it requires additional sort in local partitions before grouping. It works better for the case there are only few number of input rows in same group key for each local partition, but for the case there are numerous input rows having same group  key in local partition, doing partial aggregation can still increase the performance significantly despite additional sort.
   
   You can enable `spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition` to indicate Spark to perform partial aggregation.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org