[jira] [Commented] (FLINK-31771) Improve select available slot from SlotPool

["Lijie Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712202#comment-17712202 ] 

Lijie Wang commented on FLINK-31771:
------------------------------------

[~huwh] This makes sense to me. Besides, I think we should also reduce the number of calls of "getFreeSlotInformations". In current implementation, the scheduler will batch request slots for tasks in the same pipeline region(ExecutionSlotAllocator#allocateSlotsFor), but the slot allocator will process these requests one by one, and call "getFreeSlotInformations" once for each request. 

We can optimize it to call "getFreeSlotInformations" once for a bulk (of slot requests), instead of once for each slot request. WDYT?

> Improve select available slot from SlotPool
> -------------------------------------------
>
>                 Key: FLINK-31771
>                 URL: https://issues.apache.org/jira/browse/FLINK-31771
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>            Reporter: Weihua Hu
>            Priority: Major
>
> DefaultScheduler will request slots from SlotPool for tasks one by one.
> For each task, the PhysicalSlotProviderImpl#tryAllocateFromAvailable will retrieve all available slots from DefaultAllocatedSlotPool#getFreeSlotsInformation, and then select the best slot by SlotSelectionStrategy.
> Currently DefaultAllocatedSlotPool#getFreeSlotsInformation always calculates the taskExecutorUtilization.  This causes task schedules to be too slow when there are lots of slots, such as 20000 slots total. But only the EvenlySpreadOutLocationPreferenceSlotSelectionStrategy uses this utilization.
> So I would like to move the calculation of taskExecutorUtilization to usage. DefaultAllocatedSlotPool provides a function: getTaskExecutorUtilization, and is only used in EvenlySpreadOutLocationPreferenceSlotSelectionStrategy.
> This change could reduce the latency of allocated 20000 slots from 72s to 12s in my local IDE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)