You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Yingjie Cao (Jira)" <ji...@apache.org> on 2020/05/08 07:28:00 UTC

[jira] [Created] (FLINK-17568) Task may consume data after checkpoint barrier before performing checkpoint for unaligned checkpoint

Yingjie Cao created FLINK-17568:
-----------------------------------

             Summary: Task may consume data after checkpoint barrier before performing checkpoint for unaligned checkpoint
                 Key: FLINK-17568
                 URL: https://issues.apache.org/jira/browse/FLINK-17568
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing
    Affects Versions: 1.11.0
            Reporter: Yingjie Cao
             Fix For: 1.11.0


For unaligned checkpoint, task may consume data after the checkpoint barrier before performing checkpoint which lead to consumption of duplicated data and corruption of data stream.

More specifically, when the Netty thread notifies the checkpoint barrier for the first time and enqueue a checkpointing task in the mailbox, the task thread may still in data consumption loop and if it reads a new checkpoint barrier from another channel it will not return to the mailbox and instead it will continue to read data until a all data consumed or we have a full record, meanwhile, the data after checkpoint barrier may be read and consumed which lead to inconsistency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)