[jira] [Created] (FLINK-14162) Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies

["Zhu Zhu (Jira)" <ji...@apache.org>]

Zhu Zhu created FLINK-14162:
-------------------------------

             Summary: Unify SchedulerOperations#allocateSlotsAndDeploy implementation for all scheduling strategies
                 Key: FLINK-14162
                 URL: https://issues.apache.org/jira/browse/FLINK-14162
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
    Affects Versions: 1.10.0
            Reporter: Zhu Zhu
             Fix For: 1.10.0


In scheduler NG, scheduling strategies invokes {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to trigger scheduling of tasks.

However, {{EagerSchedulingStrategy}} and {{LazyFromSourcesSchedulingStrategy}} both invokes it by passing a batch of tasks, but requires the scheduling process to be conducted in 2 different ways:
 * {{EagerSchedulingStrategy}} requires the batch of tasks to deploy after all of them have acquired slots. This is essential to avoid partition update RPCs in streaming job scheduling.
 * {{LazyFromSourcesSchedulingStrategy}} requires tasks in the batch to allocate slot and get deployed individually, so that it can deploy a few tasks even if the slots is not enough for all tasks in the batch. This is helpful for batch job scheduling.

The scheduler then have to decide the scheduling pattern based whether the scheduling strategy is a {{LazyFromSourcesSchedulingStrategy}}. This is not good, as there can be more strategies in the future, and even customized scheduling strategies.

I think it's better to define the {{SchedulerOperations#allocateSlotsAndDeploy(Collection<ExecutionVertexDeploymentOption>)}} to be that all tasks in the batch need to be assigned and deployed together, like what we do for {{EagerSchedulingStrategy}}.
All scheduling strategies need to follow this rule. If tasks should be scheduled individually, the strategy should invoke {{allocateSlotsAndDeploy}} multiple times, one for each task. As a result, the {{LazyFromSourcesSchedulingStrategy}} needs to be adjusted for that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)