[GitHub] flink pull request #3539: [FLINK-4256] Flip1: fine gained recovery

[shuai-xu <gi...@git.apache.org>]

GitHub user shuai-xu opened a pull request:

    https://github.com/apache/flink/pull/3539

    [FLINK-4256] Flip1: fine gained recovery

    This is an informal pr for the implementation of flip1 version 1. 
    It enable that when a task fail, only restart the minimal pipelined connected executions instead of the whole execution graph.
    Main changes:
    1. ExecutionGraph doesn't manage the failover any more, it only record the finished JobVertex number and turn to FINISHED when all vertexes finish(maybe later FailoverCoordinator will take over this). Its state can only be CREATED, RUNNING, FAILED, FINISHED or SUSPENDED now.
    2. FailoverCoordinator will manage the failover now. It will generate several FailoverRegions when the EG is attached. It listens for the fail of executions. When an execution fail, it finds a FailoverRegion to finish the failover.
    3. When JM need the EG to be canceled or failed, EG will also notice FailoverCoordinator, FailoverCoordinator will notice all FailoverRegions to cancel their executions and when all executions are canceled, FailoverCoordinator will notice EG to be CANCELED or FAILED.
    4. FailoverCoordinator has server state, RUNNING, FAILING, CANCELLING, FAILED, CANCELED. 
    5. FailoverRegion contains the minimal pipelined connected executions and manager the failover of them.
    6. FailoverRegion has CREATED, RUNNING, CANCELLING, CANCELLED.
    7. One FailoverRegion may be the succeeding or preceding of others. When a preceding region failover, its all succeedings should failover too. And the succeedings should just reset its executions and wait for the preceding to start it when preceding finish. Preceding should wait for its succeedings to be CREATED and then schedule again.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shuai-xu/flink jira-4256

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3539
    
----
commit 363f1536838064edbdd5f39e41f3f19f6c511fc4
Author: shuai.xus <sh...@alibaba-inc.com>
Date:   2017-03-15T03:36:11Z

    [FLINK-4256] Flip1: fine gained recovery

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3539: Flip1: fine gained recovery

[shuai-xu <gi...@git.apache.org>]

Github user shuai-xu closed the pull request at:

    https://github.com/apache/flink/pull/3539


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3539: Flip1: fine gained recovery

[StephanEwen <gi...@git.apache.org>]

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3539
  
    @shuai-xu Had a look at the changes. In general, I like the decoupling of failover handling from the execution graph.
    
    Some suggestions:
      - How about we rename the `FailoverCoordinator` to `FailoverStrategy`?
    
      - We should probably implement one very simple failover strategy (restart all) and one better one (failover region based). That has the advantage of having a default implementation that is simple and robust (same as now) for all conservative users.
    
      - I would suggest to let commands like `fail()` and `cancel()` that go directly to the `ExecutionGraph` stay on the execution graph. They are meant to apply the whole graph and not be specially handled. The failover coordinator could only handle failures from individual tasks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3539: Flip1: fine gained recovery

[StephanEwen <gi...@git.apache.org>]

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3539
  
    Thanks for opening this! I will take a look soon...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---