[jira] [Commented] (FLINK-4256) Fine-grained recovery

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925615#comment-15925615 ] 

ASF GitHub Bot commented on FLINK-4256:
---------------------------------------

GitHub user shuai-xu opened a pull request:

    https://github.com/apache/flink/pull/3539

    [FLINK-4256] Flip1: fine gained recovery

    This is an informal pr for the implementation of flip1 version 1. 
    It enable that when a task fail, only restart the minimal pipelined connected executions instead of the whole execution graph.
    Main changes:
    1. ExecutionGraph doesn't manage the failover any more, it only record the finished JobVertex number and turn to FINISHED when all vertexes finish(maybe later FailoverCoordinator will take over this). Its state can only be CREATED, RUNNING, FAILED, FINISHED or SUSPENDED now.
    2. FailoverCoordinator will manage the failover now. It will generate several FailoverRegions when the EG is attached. It listens for the fail of executions. When an execution fail, it finds a FailoverRegion to finish the failover.
    3. When JM need the EG to be canceled or failed, EG will also notice FailoverCoordinator, FailoverCoordinator will notice all FailoverRegions to cancel their executions and when all executions are canceled, FailoverCoordinator will notice EG to be CANCELED or FAILED.
    4. FailoverCoordinator has server state, RUNNING, FAILING, CANCELLING, FAILED, CANCELED. 
    5. FailoverRegion contains the minimal pipelined connected executions and manager the failover of them.
    6. FailoverRegion has CREATED, RUNNING, CANCELLING, CANCELLED.
    7. One FailoverRegion may be the succeeding or preceding of others. When a preceding region failover, its all succeedings should failover too. And the succeedings should just reset its executions and wait for the preceding to start it when preceding finish. Preceding should wait for its succeedings to be CREATED and then schedule again.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shuai-xu/flink jira-4256

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3539
    
----
commit 363f1536838064edbdd5f39e41f3f19f6c511fc4
Author: shuai.xus <sh...@alibaba-inc.com>
Date:   2017-03-15T03:36:11Z

    [FLINK-4256] Flip1: fine gained recovery

----


> Fine-grained recovery
> ---------------------
>
>                 Key: FLINK-4256
>                 URL: https://issues.apache.org/jira/browse/FLINK-4256
>             Project: Flink
>          Issue Type: Improvement
>          Components: JobManager
>    Affects Versions: 1.1.0
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>
> When a task fails during execution, Flink currently resets the entire execution graph and triggers complete re-execution from the last completed checkpoint. This is more expensive than just re-executing the failed tasks.
> In many cases, more fine-grained recovery is possible.
> The full description and design is in the corresponding FLIP.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures
> The detail desgin for version1 is https://docs.google.com/document/d/1_PqPLA1TJgjlqz8fqnVE3YSisYBDdFsrRX_URgRSj74/edit#



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)