You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "shuai.xu (JIRA)" <ji...@apache.org> on 2019/01/10 07:36:00 UTC
[jira] [Commented] (FLINK-10433) Stepwise creation of the
ExecutionGraph sub-structures
[ https://issues.apache.org/jira/browse/FLINK-10433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739159#comment-16739159 ]
shuai.xu commented on FLINK-10433:
----------------------------------
Like this proposal. Autoscaling is very useful. Further more, we could support automatically deciding the result partition type, such as in FLINK-11299. And in the more future, we could support dynamic job graph.
> Stepwise creation of the ExecutionGraph sub-structures
> ------------------------------------------------------
>
> Key: FLINK-10433
> URL: https://issues.apache.org/jira/browse/FLINK-10433
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
> Affects Versions: 1.7.0
> Reporter: Stefan Richter
> Assignee: Stefan Richter
> Priority: Major
>
> In this step, we break the construction of ExecutionGraph substructures into multiple steps. When translating a JobGraph to an ExecutionGraph, we could only create a “skeleton” ExecutionGraph that is only built up to the ExecutionJobVertex level. This enables scheduling in two steps. First we can compute the minimal and desired amount of resources and ask the SlotPool for them. Only when the available resources match the minimal requirements, the actual scheduling would start. With this information, we can build the ExecutionVertex and Execution objects already in the right parallelism and avoid running out of resources during scheduling. This separation will help us with the autoscaling use-case, where we figure out the actual parallelism after we started with scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)