[jira] [Commented] (FLINK-14870) Introducing non-nullable assumption for task slot sharing group

["Chesnay Schepler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186436#comment-17186436 ] 

Chesnay Schepler commented on FLINK-14870:
------------------------------------------

[~trohrmann] What do you think about this?

> Introducing non-nullable assumption for task slot sharing group
> ---------------------------------------------------------------
>
>                 Key: FLINK-14870
>                 URL: https://issues.apache.org/jira/browse/FLINK-14870
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>    Affects Versions: 1.10.0
>            Reporter: Zhu Zhu
>            Priority: Major
>
> In runtime, a null slot sharing group means the vertex should be run in an individual slot which is not shared with other vertices. 
> However, at the moment, there is no way to set the slot sharing group of a vertex to null in production. This is because {{StreamGraphGenerator}} would always assign a non-null slot sharing group to an operator({{StreamNode}}), and that would be the group of the generated {{JobVertex}}. (In the case of {{DataSet}}, all vertices are in the "default" group.)
> Moreover, currently users are already able to force an operator to run in an individual slot by specify a different slot sharing group key to that operator.
> So looks that supporting null slot sharing groups in runtime does not make much sense. And it's making things more complex, with codes like {{@Nullable}} annotations for slot sharing group, task required resources calculations, and single task slot allocation. 
> To make runtime code base and future development simpler, I'd propose to introduce the assumption that task slot sharing group should not be null. 
> The detailed proposal is:
> 1. In {{StreamingJobGraphGenerator}}, we check to ensure the slot sharing group key of a {{StreamNode}} is non-null, so that we always set a non-null {{SlotSharingGroup}} to a JobVertex
> 2. Remove all the {{@Nullable}} annotations for {{SlotSharingGroup}} and {{SlotSharingGroupID}} in several classes
> 3. Always use the slot sharing group resources as the physical slot resources for a task
> 4. Remove {{SchedulerImp#allocateSingleSlot}} and the related code paths
> cc [~chesnay] [~gjy] [~trohrmann]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)