[jira] [Created] (FLINK-22502) DefaultCompletedCheckpointStore drops unrecoverable checkpoints silently

["Till Rohrmann (Jira)" <ji...@apache.org>]

Till Rohrmann created FLINK-22502:
-------------------------------------

             Summary: DefaultCompletedCheckpointStore drops unrecoverable checkpoints silently
                 Key: FLINK-22502
                 URL: https://issues.apache.org/jira/browse/FLINK-22502
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing, Runtime / Coordination
    Affects Versions: 1.12.2, 1.11.3, 1.13.0, 1.14.0
            Reporter: Till Rohrmann
             Fix For: 1.14.0, 1.13.1, 1.12.4


The {{DefaultCompletedCheckpointStore.recover()}} tries to be resilient if it cannot recover a checkpoint (e.g. due to a transient storage outage or a checkpoint being corrupted). This behaviour was introduced with FLINK-7783.

The problem is that this behaviour might cause us to ignore the latest valid checkpoint if there is a transient problem when restoring it. This might be ok for at least once processing guarantees, but it clearly violates exactly once processing guarantees. On top of it, it is very hard to spot.

I propose to change this behaviour so that {{DefaultCompletedCheckpointStore.recover()}} fails if it cannot read the checkpoints it is supposed to read. If the {{recover}} method fails during a recovery, it will kill the process. This will usually restart the process which will retry the checkpoint recover operation. If the problem is of transient nature, then it should eventually succeed. In case that this problem occurs during an initial job submission, then the job will directly transition to a {{FAILED}} state.

The proposed behaviour entails that if there is a permanent problem with the checkpoint (e.g. corrupted checkpoint), then Flink won't be able to recover without the intervention of the user. I believe that this is the right decision because Flink can no longer give exactly once guarantees in this situation and a user needs to explicitly resolve this situation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)