You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Shuo Cheng (JIRA)" <ji...@apache.org> on 2018/11/21 03:17:00 UTC

[jira] [Comment Edited] (FLINK-5601) Window operator does not checkpoint watermarks

    [ https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694096#comment-16694096 ] 

Shuo Cheng edited comment on FLINK-5601 at 11/21/18 3:16 AM:
-------------------------------------------------------------

Hi, JiaYi, You say that the watermark restored is the lowest in the union state, but actually, I find that you use the highest watermark in the code, is that what you mean here? 


was (Author: icshuo):
Hi, JiaYi, You say that the watermark restored is the lowest in the union state, but actually, I find that you use the highest watermark in the code? 

> Window operator does not checkpoint watermarks
> ----------------------------------------------
>
>                 Key: FLINK-5601
>                 URL: https://issues.apache.org/jira/browse/FLINK-5601
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0
>            Reporter: Ufuk Celebi
>            Assignee: Jiayi Liao
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.8.0
>
>
> During release testing [~stefanrichter83@gmail.com] and I noticed that watermarks are not checkpointed in the window operator.
> This can lead to non determinism when restoring checkpoints. I was running an adjusted {{SessionWindowITCase}} via Kafka for testing migration and rescaling and ran into failures, because the data generator required determinisitic behaviour.
> What happened was that on restore it could happen that late elements were not dropped, because the watermarks needed to be re-established after restore first.
> [~aljoscha] Do you know whether there is a special reason for explicitly not checkpointing watermarks?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)