[GitHub] flink issue #3340: [FLINK-5703] [Job Manager] ExecutionGraph recovery via re...

[zhijiangW <gi...@git.apache.org>]

Github user zhijiangW commented on the issue:

    https://github.com/apache/flink/pull/3340
  
    @StephanEwen , sorry for delay update of this feature.
    
    This modification covers almost all the work on job manager side concerning about the suggestions you mentioned before, including:
    
    1. Once `JobManagerRunner` granting leadership, it will check the job status by `RunningJobsRegistry`. If the job is already running, then the `JobManager` will enter reconciling process and confirm leadership last.
    To do so, we can confirm that the `TaskManager` will not report task status before `JobManager` reconciles, and it is also reasonable for 'JobManager` exposing its address to outside `TaskManager` after internal initialization. To keep the behavior consistency, I modified the `EmbeddedLeaderService` to set leader flag as true in the process of `grantLeadership` instead of 'confirmLeadership`.
    
    2. 'ExecutionGraph' recovery based on task status report form `TaskManager`, and I created the new execution attempt to replace the current one for recovery, also the same case for `IntermediateResultPartition` to avoid updating the final fields directly in previous way. And also integrating two phases of recovery into one phase you mentioned before.
    
    3. After duration time, the `ExecutionGraph` decides to enter `RUNNING` or `FAILING` from `RECONCILING` based on reconciliation results.
    
    Currently I have not considered the improvements for `LAZY_FROM_SOURCES` schedule mode in this version, because it needs to remember how many vertexes have already been scheduled before `JobManager` failure. Maybe we can improve it in next version if needed.
    
    I may supplement some extra tests and update the attached design doc after you approve this realization, and the 'TaskManager' part will be submitted in another separate PR which mainly includes:
    
    1. `TaskManager` will not fail task directly after notified of `JobManager` leader lost, and triggers a timer to later check whether reconnects to the new job leader successfully.
    
    2. If reconnection success, it will report the current internal task and slot status that belongs to this job leader, otherwise it will fail the tasks as the current behavior.
    
    Wish your further response, and I will begin to consider resource issues of FLIP-6, thank you!
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---