[jira] [Updated] (FLINK-18405) Add watermark support for unaligned checkpoints

["Yun Gao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yun Gao updated FLINK-18405:
----------------------------
    Fix Version/s: 1.16.0

> Add watermark support for unaligned checkpoints
> -----------------------------------------------
>
>                 Key: FLINK-18405
>                 URL: https://issues.apache.org/jira/browse/FLINK-18405
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>    Affects Versions: 1.12.0
>            Reporter: Arvid Heise
>            Priority: Not a Priority
>              Labels: auto-deprioritized-major
>             Fix For: 1.15.0, 1.16.0
>
>
> Currently, Flink generates the watermark as a first step of recovery instead of 
> storing the latest watermark in the operators to ease rescaling. In unaligned 
> checkpoints, that means on recovery, Flink generates watermarks after it 
> restores in-flight data. If your pipeline uses an operator that applies the
> latest watermark on each record, it will produce incorrect results during 
> recovery if the watermark is not directly or indirectly part of the operator 
> state. Thus, SQL OVER operator should not be used with unaligned
> checkpoints, while window operators are safe to use. 
> A possible solution is to store the watermark in the operator state. If rescaling may occur, watermarks should be stored per key-group in a union-state. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)