You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Andrey Zagrebin (Jira)" <ji...@apache.org> on 2020/08/05 16:33:00 UTC
[jira] [Updated] (FLINK-18751) Implement
SlotSharingExecutionSlotAllocator
[ https://issues.apache.org/jira/browse/FLINK-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrey Zagrebin updated FLINK-18751:
------------------------------------
Description:
SlotSharingExecutionSlotAllocator maintains a SharedSlot for each ExecutionSlotSharingGroup. SlotSharingExecutionSlotAllocator allocates physical slots for SharedSlot(s) and then allocates logical slots from it for scheduled tasks. In this way, the slot sharing hints can be respected in the ExecutionSlotAllocator. And we no longer need to rely on the SlotProvider to do the slot sharing matching. Co-location constraints will be respected since co-located subtasks will be in the same ExecutionSlotSharingGroup.
The physical slot will be lazily allocated for a SharedSlot, upon any hosted subtask asking for the SharedSlot. Each subsequent sharing subtask allocates a logical slot from the SharedSlot. The SharedSlot/physical slot can be released only if all the requested logical slots are released or canceled.
h4. Slot Allocation Process
When SlotSharingExecutionSlotAllocator receives a set of tasks to allocate slots for, it should do the following:
# Map the tasks to ExecutionSlotSharingGroup(s)
# Check which ExecutionSlotSharingGroup(s) _already have_ SharedSlot(s)
# For all involved ExecutionSlotSharingGroup(s) _which do not have a SharedSlot_ yet:
## Create a SlotProfile future by MergingSharedSlotProfileRetriever and then
## Allocate a physical slot from the PhysicalSlotProvider
## Create SharedSlot based on the returned physical slot futures
# Allocate logical slot futures for the tasks from all corresponding SharedSlot(s).
# If physical slot future fails, cancel its pending logical slot requests within the SharedSlot
# Generates SlotExecutionVertexAssignment(s) based on the logical slot futures and returns the results.
was:
SlotSharingExecutionSlotAllocator maintains a SharedSlot for each ExecutionSlotSharingGroup. SlotSharingExecutionSlotAllocator allocates physical slots for SharedSlot(s) and then allocates logical slots from it for scheduled tasks. In this way, the slot sharing hints can be respected in the ExecutionSlotAllocator. And we no longer need to rely on the SlotProvider to do the slot sharing matching. Co-location constraints will be respected since co-located subtasks will be in the same ExecutionSlotSharingGroup.
The physical slot will be lazily allocated for a SharedSlot, upon any hosted subtask asking for the SharedSlot. Each subsequent sharing subtask allocates a logical slot from the SharedSlot. The SharedSlot/physical slot can be released only if all the requested logical slots are released or canceled.
h4. Slot Allocation Process
When SlotSharingExecutionSlotAllocator receives a set of tasks to allocate slots for, it should do the following:
# Map the tasks to ExecutionSlotSharingGroup(s)
# Check which ExecutionSlotSharingGroup(s) _already have_ SharedSlot(s)
# For all involved ExecutionSlotSharingGroup(s) _which do not have a SharedSlot_ yet:
## Create a SlotProfile future by MergingSharedSlotProfileRetriever and then
## Allocate a physical slot from the PhysicalSlotProvider
## Create SharedSlot based on the returned physical slot futures
## If physical slot future fails, remove the SharedSlot
# Allocate logical slot futures for the tasks from all corresponding SharedSlot(s).
# Generates SlotExecutionVertexAssignment(s) based on the logical slot futures and returns the results.
> Implement SlotSharingExecutionSlotAllocator
> -------------------------------------------
>
> Key: FLINK-18751
> URL: https://issues.apache.org/jira/browse/FLINK-18751
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
>
> SlotSharingExecutionSlotAllocator maintains a SharedSlot for each ExecutionSlotSharingGroup. SlotSharingExecutionSlotAllocator allocates physical slots for SharedSlot(s) and then allocates logical slots from it for scheduled tasks. In this way, the slot sharing hints can be respected in the ExecutionSlotAllocator. And we no longer need to rely on the SlotProvider to do the slot sharing matching. Co-location constraints will be respected since co-located subtasks will be in the same ExecutionSlotSharingGroup.
> The physical slot will be lazily allocated for a SharedSlot, upon any hosted subtask asking for the SharedSlot. Each subsequent sharing subtask allocates a logical slot from the SharedSlot. The SharedSlot/physical slot can be released only if all the requested logical slots are released or canceled.
> h4. Slot Allocation Process
> When SlotSharingExecutionSlotAllocator receives a set of tasks to allocate slots for, it should do the following:
> # Map the tasks to ExecutionSlotSharingGroup(s)
> # Check which ExecutionSlotSharingGroup(s) _already have_ SharedSlot(s)
> # For all involved ExecutionSlotSharingGroup(s) _which do not have a SharedSlot_ yet:
> ## Create a SlotProfile future by MergingSharedSlotProfileRetriever and then
> ## Allocate a physical slot from the PhysicalSlotProvider
> ## Create SharedSlot based on the returned physical slot futures
> # Allocate logical slot futures for the tasks from all corresponding SharedSlot(s).
> # If physical slot future fails, cancel its pending logical slot requests within the SharedSlot
> # Generates SlotExecutionVertexAssignment(s) based on the logical slot futures and returns the results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)