[jira] [Assigned] (FLINK-6626) Unifying lifecycle management of SubmittedJobGraph- and CompletedCheckpointStore

["Gary Yao (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gary Yao reassigned FLINK-6626:
-------------------------------

    Assignee: Tarush Grover

> Unifying lifecycle management of SubmittedJobGraph- and CompletedCheckpointStore
> --------------------------------------------------------------------------------
>
>                 Key: FLINK-6626
>                 URL: https://issues.apache.org/jira/browse/FLINK-6626
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Till Rohrmann
>            Assignee: Tarush Grover
>            Priority: Major
>
> Currently, Flink uses the {{SubmittedJobGraphStore}} to persist {{JobGraphs}} such that they can be recovered in case of failures. The {{SubmittedJobGraphStore}} is managed by by the {{JobManager}}. Additionally, Flink has the {{CompletedCheckpointStore}} which stores checkpoints for a given {{ExecutionGraph}}/job. The {{CompletedCheckpointStore}} is managed by the {{CheckpointCoordinator}}.
> The {{SubmittedJobGraphStore}} and the {{CompletedCheckpointStore}} are somewhat related because in the latter we store checkpoints for jobs contained in the former. I think it would be nice wrt lifecycle management to let the {{SubmittedJobGraphStore}} manage the lifecycle of the {{CompletedCheckpointStore}}, because often it does not make much sense to keep only checkpoints without a job or a job without checkpoints. 
> An idea would be when we register a job with the {{SubmittedJobGraphStore}} then it returns a {{CompletedCheckpointStore}}. This store can then be given to the {{CheckpointCoordinator}} to store the checkpoints. When a job enters a terminal state it could be the responsibility of the {{SubmittedJobGraphStore}} to decide what to do with the job data ({{JobGraph}} and {{Checkpoints}}), e.g. keeping it or cleaning it up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)