[jira] [Created] (FLINK-10433) Stepwise creation of the ExecutionGraph sub-structures

["Stefan Richter (JIRA)" <ji...@apache.org>]

Stefan Richter created FLINK-10433:
--------------------------------------

             Summary: Stepwise creation of the ExecutionGraph sub-structures
                 Key: FLINK-10433
                 URL: https://issues.apache.org/jira/browse/FLINK-10433
             Project: Flink
          Issue Type: Sub-task
          Components: Distributed Coordination
    Affects Versions: 1.7.0
            Reporter: Stefan Richter
            Assignee: Stefan Richter


In this step, we break the construction of ExecutionGraph substructures into multiple steps. When translating a JobGraph to an ExecutionGraph, we could only create a “skeleton” ExecutionGraph that is only built up to the ExecutionJobVertex level. This enables scheduling in two steps. First we can compute the minimal and desired amount of resources and ask the SlotPool for them. Only when the available resources match the minimal requirements, the actual scheduling would start. With this information, we can build the ExecutionVertex and Execution objects already in the right parallelism and avoid running out of resources during scheduling. This separation will help us with the autoscaling use-case, where we figure out the actual parallelism after we started with scheduling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)