[GitHub] jose-torres commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode

[GitBox <gi...@apache.org>]

jose-torres commented on issue #23576: [SPARK-26655] [SS] Support multiple aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-461530739
 
 
   I'd agree that min is the only reasonable way to compute an operator watermark. What I think we need a design for is operator watermarks in general, and how they slot into the rest of Spark. Questions I worry can't be addressed by a PR include:
   
   * I have a plan tree A: EventTimeExec -> B: StatefulOperator -> C: StatefulOperator. Can C use the watermark in A? If so, is it safe to do that when B transforms or projects away the watermarked column - if not, what are the rules for how watermarks must be provided with multiple aggregates?
   * Do all of our optimization and execution rules respect the semantics of operator watermarks?
   * We can currently call `withWatermark` at any point in the query plan. Is this consistent with operator watermarks? Even if we can support the two of them together, do we want to?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org