[jira] [Commented] (FLINK-13452) Pipelined region failover strategy does not recover Job if checkpoint cannot be read

["Yun Tang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896372#comment-16896372 ] 

Yun Tang commented on FLINK-13452:
----------------------------------

Before FLINK-13060, the process to restart tasks within {{AdaptedRestartPipelinedRegionStrategyNG}} is:
{code:java}
cancelTasks --> resetTasks --> handleExcpetion
{code}
After FLINK-13060, the process changed to:
{code:java}
                   resetTasks
                       |^|
                       | |
cancelTasks --> strategy restart --> handleExcpetion                       
{code}
The bug happened due to the future task of {{resetTasks}} failed but die silently. And as you can see, this bug has no direct relationship between region failover to load checkpointed state.

Previously I try to catch any exception within {{resetTasks}} itself to {{failGlobal}} in my [PR-9268|https://github.com/apache/flink/pull/9268]. However, this would meet another problem if {{failGlobal}} come across an exception and cannot be caught by {{FatalExitExceptionHandler}}.

I still have not found any graceful solution to this bug, [~gjy], [~Zentol] do you have any suggestions for this?

 

> Pipelined region failover strategy does not recover Job if checkpoint cannot be read
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-13452
>                 URL: https://issues.apache.org/jira/browse/FLINK-13452
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 1.9.0, 1.10.0
>            Reporter: Gary Yao
>            Assignee: Yun Tang
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.9.0
>
>         Attachments: jobmanager.log
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The job does not recover if a checkpoint cannot be read and {{jobmanager.execution.failover-strategy}} is set to _"region"_. 
> *Analysis*
> The {{RestartCallback}} created by {{AdaptedRestartPipelinedRegionStrategyNG}} throws a \{{RuntimeException}} if no checkpoints could be read. When the restart is invoked in a separate thread pool, the exception is swallowed. See:
> [https://github.com/apache/flink/blob/21621fbcde534969b748f21e9f8983e3f4e0fb1d/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/AdaptedRestartPipelinedRegionStrategyNG.java#L117-L119]
> [https://github.com/apache/flink/blob/21621fbcde534969b748f21e9f8983e3f4e0fb1d/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/restart/FixedDelayRestartStrategy.java#L65]
> *Expected behavior*
>  * Job should restart
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)