[jira] [Commented] (FLINK-12122) Spread out tasks evenly across all available registered TaskManagers

["YufeiLiu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011666#comment-17011666 ] 

YufeiLiu commented on FLINK-12122:
----------------------------------

[~trohrmann] I noticed that this feature is evenly across available "registered" TaskManagers, so it's might not work with on yarn mode? 
I write a test, each container has 2 slot : 
{code:java}
source.setParallelism(2).map().setParallelism(4).addsink().setParallelism(2)
{code}
In first time schedule, all source subtask are in one TM, but sink subtasks are spread out. ResourceManager allocate container after request slot,  so this config doesn't work with source operator. 
Is ther any way to spread out all tasks on yarn?

> Spread out tasks evenly across all available registered TaskManagers
> --------------------------------------------------------------------
>
>                 Key: FLINK-12122
>                 URL: https://issues.apache.org/jira/browse/FLINK-12122
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>    Affects Versions: 1.6.4, 1.7.2, 1.8.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.9.2, 1.10.0
>
>         Attachments: image-2019-05-21-12-28-29-538.png, image-2019-05-21-13-02-50-251.png
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> With Flip-6, we changed the default behaviour how slots are assigned to {{TaskManages}}. Instead of evenly spreading it out over all registered {{TaskManagers}}, we randomly pick slots from {{TaskManagers}} with a tendency to first fill up a TM before using another one. This is a regression wrt the pre Flip-6 code.
> I suggest to change the behaviour so that we try to evenly distribute slots across all available {{TaskManagers}} by considering how many of their slots are already allocated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)