[GitHub] [spark] viirya commented on pull request #33689: [SPARK-36463][SS] Prohibit update mode in streaming aggregation with session window

[GitBox <gi...@apache.org>]

viirya commented on pull request #33689:
URL: https://github.com/apache/spark/pull/33689#issuecomment-896287697


   > This doesn't hold true for session window based streaming aggregation. If you're trying to upsert the output based on the grouping key, it's high likely possible that existing row is not updated (overwritten) and ended up with having different rows.
   
   To be clear, I looked the current approach of UPDATE mode. We considers the row is updated if there is no existing row with the state key (session key + session start time), or the stored value isn't the same as the current value that will be stored.
   
   But it is also possible that the session window was extended backward at the session start time. So the updated rows are not actually accurate.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org