[jira] [Created] (FLINK-18405) Add watermark support for unaligned checkpoints

["Arvid Heise (Jira)" <ji...@apache.org>]

Arvid Heise created FLINK-18405:
-----------------------------------

             Summary: Add watermark support for unaligned checkpoints
                 Key: FLINK-18405
                 URL: https://issues.apache.org/jira/browse/FLINK-18405
             Project: Flink
          Issue Type: Improvement
    Affects Versions: 1.12.0
            Reporter: Arvid Heise


Currently, Flink generates the watermark as a first step of recovery instead of 
storing the latest watermark in the operators to ease rescaling. In unaligned 
checkpoints, that means on recovery, Flink generates watermarks after it 
restores in-flight data. If your pipeline uses an operator that applies the
latest watermark on each record, it will produce incorrect results during 
recovery if the watermark is not directly or indirectly part of the operator 
state. Thus, SQL OVER operator should not be used with unaligned
checkpoints, while window operators are safe to use. 

A possible solution is to store the watermark in the operator state. If rescaling may occur, watermarks should be stored per key-group in a union-state. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)