[jira] [Commented] (FLINK-19714) Assign different logical pipelined regions of a DataSet job to different slot sharing group

["Zhu Zhu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216682#comment-17216682 ] 

Zhu Zhu commented on FLINK-19714:
---------------------------------

cc [~trohrmann] [~azagrebin]

> Assign different logical pipelined regions of a DataSet job to different slot sharing group
> -------------------------------------------------------------------------------------------
>
>                 Key: FLINK-19714
>                 URL: https://issues.apache.org/jira/browse/FLINK-19714
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>    Affects Versions: 1.12.0
>            Reporter: Zhu Zhu
>            Priority: Major
>             Fix For: 1.12.0
>
>
> Currently, job vertices of a DataSet job will be put into the same SlotSharingGroup. It does not make sense because producer and consumer pipelined regions will not be able to share slots. This can lead to a waste of managed memory.
> We should assign different logical pipelined regions of a DataSet job to different slot sharing group, like how we does for Blink planner batch jobs. Then we can refine the managed memory fraction calculation for DataSet operators, i.e. TaskConfig.MEMORY_DRIVER and TaskConfig.MEMORY_INPUT_PREFIX.*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)