[jira] [Commented] (FLINK-5703) ExecutionGraph recovery based on reconciliation with TaskManager reports

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871499#comment-15871499 ] 

ASF GitHub Bot commented on FLINK-5703:
---------------------------------------

GitHub user wangzhijiang999 opened a pull request:

    https://github.com/apache/flink/pull/3340

    [FLINK-5703][runtime]Job manager failure recovery via reconciliation with TaskManager reports

    This is part of [Non-disruptive JobManager Failures via Reconciliation ](https://issues.apache.org/jira/browse/FLINK-4911). 
    
    The design doc for this part is attached in the JIRA and it mainly contains the following work:
    
    - **RECONCILING** state transition for job status and execution.
    - The data structure recovery for **ExecutionGraph**, **ExecutionVertex**,**Execution** based on **TaskManager** reports. 
    
    Two parts are left for the whole feature:
    
    - The related modifications in **TaskManger** side will be submitted in another JIRA, including not cancel the tasks when notified job leader changed and report the task status to **JobManager**, etc.
    - This PR will not make an effect on the current master branch, so it is safe to merge. The reconcile logic should be triggered by **JobManagerRunner** when grants leadership, but it is dependent on [Determine whether the job starts from last JobManager failure](https://issues.apache.org/jira/browse/FLINK-5501) , so I will modify the **JobManagerRunner** logic after [FLINK-5501](https://issues.apache.org/jira/browse/FLINK-5501) is merged.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wangzhijiang999/flink FLINK-5703

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3340
    
----
commit 7337754bd8d12808a328b55e184b0fe59cf6e11d
Author: 淘江 <ta...@alibaba-inc.com>
Date:   2017-02-17T08:41:44Z

    [FLINK-5703][runtime]Job manager failure recovery based on reconciliation with TaskManager reports

----


> ExecutionGraph recovery based on reconciliation with TaskManager reports
> ------------------------------------------------------------------------
>
>                 Key: FLINK-5703
>                 URL: https://issues.apache.org/jira/browse/FLINK-5703
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Coordination, JobManager
>            Reporter: Zhijiang Wang
>            Assignee: Zhijiang Wang
>
> The ExecutionGraph structure would be recovered from TaskManager reports during reconciling period, and the necessary information includes:
>     - Execution: ExecutionAttemptID, AttemptNumber, StartTimestamp, ExecutionState, SimpleSlot, PartialInputChannelDeploymentDescriptor(Consumer Execution)
>     - ExecutionVertex: Map<IntermediateResultPartitionID, IntermediateResultPartition>
>     - ExecutionGraph: ConcurrentHashMap<ExecutionAttemptID, Execution>
> For {{RECONCILING}} ExecutionState, it should be transition into any existing task states ({{RUNNING}},{{CANCELED}},{{FAILED}},{{FINISHED}}). To do so, the TaskManger should maintain the terminal task state ({{CANCELED}},{{FAILED}},{{FINISHED}}) for a while and we try to realize this mechanism in another jira. In addition, the state transition would trigger different actions, and some actions rely on above necessary information. Considering this limit, the recovery process will be divided into two steps:
>     - First, recovery all other necessary information except ExecutionState.
>     - Second, transition ExecutionState into real task state and trigger actions. The behavior is the same with current {{UpdateTaskExecutorState}}.
> To make logic easy and consistency, during recovery period, all the other RPC messages ({{UpdateTaskExecutionState}}, {{ScheduleOrUpdateConsumers}},etc) from TaskManager should be refused temporarily and responded with a special message by JobMaster. Then the TaskManager should retry to send these messages later until JobManager ends recovery and acknowledgement.
> For {{RECONCILING}} JobStatus, it would be transition into one of the states ({{RUNNING}},{{FAILING}},{{FINISHED}}) after recovery.
>     - {{RECONCILING}} to {{RUNNING}}: All the TaskManager report within duration time and all the tasks are in {{RUNNING}} states.
>     - {{RECONCILING}} to {{FAILING}}: One of the TaskManager does not report in time, or one of the tasks state is in {{FAILED}} or {{CANCELED}}
>     - {{RECONCILING}} to {{FINISHED}}: All the TaskManger report within duration time and all the tasks are in {{FINISHED}} states.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)