[GitHub] [spark] HeartSaVioR edited a comment on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode

[GitBox <gi...@apache.org>]

HeartSaVioR edited a comment on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-496409067
 
 
   I'm trying hard to think about the cases which this proposal can bring correctness issue, and I found nothing. Please correct me if I'm mistaken here.
   
   First of all, Spark doesn't support late firing - Spark includes `delay` in watermark instead of dealing them separately, so the possible latest fire for specific window would be when watermark passes by the window. In other words, once watermark passes by specific window, the window should be evicted (and might be also fired) and never be sent in further batches.
   
   (If you may indicate, yes, "complete mode" breaks this. But you know, "complete mode" just ignores watermark and only does upsert so irrelevant with watermark in any way, and not suitable on stateful query. I think "complete mode" is not the one worth to discuss about "logically correct" - we should just drop out.)
   
   Second, if I'm not missing anything, all of these approaches we are considering as options ensure that watermark for (N+1) level of stage cannot be higher than watermarks for N level of input stages.
   
   Considering two conditions together, these conditions ensure that (N+1) level of stage never misses the outputs from N level of stage. This should apply to all modes except complete mode.
   
   Let's apply this to update mode. You can get early firing outputs from N level of stages in (N+1) level of stage (and retraction is needed to handle these outputs correctly), but watermark for (N+1) stage will not be advanced anyway if we ensure condition 2 in above, so it doesn't break watermark for (N+1) stage.
   
   What do you think? Does my explanation sound good to convince us?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org