You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Zhu Zhu (Jira)" <ji...@apache.org> on 2022/06/20 11:43:00 UTC

[jira] [Created] (FLINK-28137) Introduce SpeculativeScheduler

Zhu Zhu created FLINK-28137:
-------------------------------

             Summary: Introduce SpeculativeScheduler
                 Key: FLINK-28137
                 URL: https://issues.apache.org/jira/browse/FLINK-28137
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
            Reporter: Zhu Zhu
             Fix For: 1.16.0


A SpeculativeScheduler will be used if speculative execution is enabled. It extends AdaptiveBatchScheduler so that speculative execution can work along with the feature to adaptively tuning parallelisms for batch jobs.

The major differences of SpeculativeScheduler are:
 * SpeculativeScheduler needs to be able to directly deploy an Execution, while AdaptiveBatchScheduler can only perform ExecutionVertex level deployment.
 * SpeculativeScheduler does not restart the ExecutionVertex if an execution fails when any other current execution is still making progress
 * SpeculativeScheduler listens on slow tasks. Once there are slow tasks, it will block the slow nodes and deploy speculative executions of the slow tasks on other nodes.
 * Once any execution finishes, SpeculativeScheduler will cancel all the remaining executions of the same execution vertex.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)