[jira] [Assigned] (FLINK-14162) Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies

["Zhu Zhu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhu Zhu reassigned FLINK-14162:
-------------------------------

    Assignee: Zhu Zhu

> Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-14162
>                 URL: https://issues.apache.org/jira/browse/FLINK-14162
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>    Affects Versions: 1.10.0
>            Reporter: Zhu Zhu
>            Assignee: Zhu Zhu
>            Priority: Major
>             Fix For: 1.11.0
>
>
> In scheduler NG, scheduling strategies invokes {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to trigger scheduling of tasks.
> However, {{EagerSchedulingStrategy}} and {{LazyFromSourcesSchedulingStrategy}} both invokes it by passing a batch of tasks, but requires the scheduling process to be conducted in 2 different ways:
>  * {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after all of them have acquired slots. This is essential to avoid partition update RPCs in streaming job scheduling.
>  * {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to allocate slot and get deployed individually, so that it can deploy a few tasks even if the slots is not enough for all tasks in the batch. This is helpful for batch job scheduling.
> The scheduler then have to decide the scheduling pattern based whether the scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not good, as there can be more strategies in the future, and even customized scheduling strategies.
> I think it's better to define the {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to be that all tasks in the batch need to be assigned and deployed together, like what we do for {{EagerSchedulingStrategy}}.
> All scheduling strategies need to follow this rule. If tasks should be scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}} multiple times, one for each task. As a result, the {{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)