[jira] [Commented] (YARN-2989) Better Load Balancing in Fair Scheduler

["Kannan Rajah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257576#comment-14257576 ] 

Kannan Rajah commented on YARN-2989:
------------------------------------

Just looked at YARN-1290. Since it is doing continuous scheduling, and it is decoupled from hearbeat, it is essentially doing the load balancing that was done in MR1.
I will take a closer look to see if it satisfies all use cases and close this bug if its no more improvement is needed.

> Better Load Balancing in Fair Scheduler
> ---------------------------------------
>
>                 Key: YARN-2989
>                 URL: https://issues.apache.org/jira/browse/YARN-2989
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.5.0
>            Reporter: Kannan Rajah
>
> While porting Fair Scheduler from MR1, we seem to have changed the logic behind task distribution across nodes (MAPREDUCE-3451).
> In MR1, a load factor was computed using runnableMaps/totalMapSlots and this was used to determine how many tasks need to be given to a node such that the overall cluster load is evenly distributed. In one heartbeat, we could assign multiple tasks. In YARN, we have the option to assign multiple tasks to a node, but this is disabled by default (YARN-302). Even when it is enabled, the number of tasks to assign is statically configured. So it won't ensure that load is evenly distributed. Why not bring back the load factor based check? Any reason why it was not done? This is actually more relevant with label based scheduling.
> If there are no objections, I would like to implement it for both normal and label based scheduling scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)