[jira] [Commented] (FLINK-9260) Introduce a friendly way to resume the job from externalized checkpoints automatically

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456651#comment-16456651 ] 

Stephan Ewen commented on FLINK-9260:
-------------------------------------

I think the most important step to get right is is to add functionality to the {{CheckpointStorage}} to find the latest completed checkpoint.

We may need to enhance that one a bit, in order to write either a success marker file, or write the metadata file first to a temp file and then atomically rename it (with an equivalent workaround for S3).

> Introduce a friendly way to resume the job from externalized checkpoints automatically
> --------------------------------------------------------------------------------------
>
>                 Key: FLINK-9260
>                 URL: https://issues.apache.org/jira/browse/FLINK-9260
>             Project: Flink
>          Issue Type: Improvement
>          Components: Client
>    Affects Versions: 1.5.0
>            Reporter: Sihua Zhou
>            Assignee: Sihua Zhou
>            Priority: Major
>             Fix For: 1.6.0
>
>
> Currently, it's quite a bit not friendly for users to recover job from the externalized checkpoint, user need to find the dedicate dir for the job which is not a easy thing when there are too many jobs. This ticket attend to introduce a more friendly way to allow the user to use the externalized checkpoint to do recovery.
> The implementation steps are copied from the comments of [~StephanEwen] in [9043|https://issues.apache.org/jira/browse/FLINK-9043]
> - We could make this an option where you pass a flag (-r) to automatically look for the latest checkpoint in a given directory.
> - If more than one jobs checkpointed there before, this operation would fail.
> - We might also need a way to have jobs not create the UUID subdirectory, otherwise the scanning for the latest checkpoint would not easily work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)