[jira] [Commented] (CALCITE-3509) Watermark Propagation

["Rui Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020527#comment-17020527 ] 

Rui Wang commented on CALCITE-3509:
-----------------------------------

I am trying to proposal something about EMIT syntax in dev@. If that idea looks good then watermark propagation will become an implementation detail that engines will need to implement their own version of "current_watermark()" function, which will make this Jira very trivial. 

> Watermark Propagation 
> ----------------------
>
>                 Key: CALCITE-3509
>                 URL: https://issues.apache.org/jira/browse/CALCITE-3509
>             Project: Calcite
>          Issue Type: Sub-task
>            Reporter: Rui Wang
>            Priority: Major
>
> To have a useful enumerable implementation for streaming windowing semantic in Calcite,  and also make Emit syntax useful, we will need to introduce watermark into Calcite. More specifically, the following should be discussed:
> 1. How is watermark generated? What changes are required on source/table scan?
> 2. How does watermark propagate through the whole plan? How are watermarks from multiple inputs aggregated together?  
> 3. Where watermark is used to control the time to emit data? 
> 4. How are event timestamp combined per key (typical in aggregation/join).
> From Julian:
> 0. Can watermarks be handled using algebraic rewrites? I.e. can we use the usual operators but restart them, giving them just the rows that changed, when watermarks arrive.
> This is consistent with Calcite's general approach, which is to solve things at a high level using algebra rather than dive into the details of a particular kind of engine.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)