[jira] [Comment Edited] (FLINK-11843) Dispatcher fails to recover jobs if leader change happens during JobManagerRunner termination

["TisonKun (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887694#comment-16887694 ] 

TisonKun edited comment on FLINK-11843 at 7/23/19 12:58 AM:
------------------------------------------------------------

[~till.rohrmann] in the attached mail you said "As a consequence the created future from the 2nd leadership will never be executed."

I suspect that the previous main thread executor was invalidated but still the created future would be executed but failed for a mismatch fencing token? In this case there would be an error instead of stuck. I'm curious why the future was never executed instead of executed and produced an error.(UPDATE: gotcha that we schedule the start future in the MainThreadExecutor and if leader changes, the action failed but not exception returned because it is a {{tell}} and thus the future never completed)

[~stevenz3wu] could you please share the log here or send to me?


was (Author: tison):
[~till.rohrmann] in the attached mail you said "As a consequence the created future from the 2nd leadership will never be executed."

I suspect that the previous main thread executor was invalidated but still the created future would be executed but failed for a mismatch fencing token? In this case there would be an error instead of stuck. I'm curious why the future was never executed instead of executed and produced an error.(UPDATE: gotcha that we schedule the start future in the MainThreadExecutor and if leader changes, the action failed but not exception returned because it is a {tell} and thus the future never completed)

[~stevenz3wu] could you please share the log here or send to me?

> Dispatcher fails to recover jobs if leader change happens during JobManagerRunner termination
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-11843
>                 URL: https://issues.apache.org/jira/browse/FLINK-11843
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 1.7.2, 1.8.0, 1.9.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Critical
>             Fix For: 1.9.0, 1.10.0
>
>
> The {{Dispatcher}} fails to recover jobs if a leader change happens during the {{JobManagerRunner}} termination of the previous run. The problem is that we schedule the start future of the recovered {{JobGraph}} using the {{MainThreadExecutor}} and additionally require that this future is completed before any other recovery operation from a subsequent leadership session is executed. If now the leadership changes, the {{MainThreadExecutor}} will be invalidated and the scheduled future will never be completed.
> The relevant ML thread: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/1-7-1-job-stuck-in-suspended-state-td26439.html



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)