[jira] Commented: (MAPREDUCE-1198) Alternatively schedule different types of tasks in fair share scheduler

["Matei Zaharia (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778540#action_12778540 ] 

Matei Zaharia commented on MAPREDUCE-1198:
------------------------------------------

Hi Scott,

I don't think what you have is quite the right fix, because when a node has space to launch more than one task, the scheduler will still assign multiple tasks of the same type rather than alternating between them. In the pre-0.21 fair scheduler, where only one task of each type was assigned per heartbeat, this solution would've worked okay. However, in the current trunk, the scheduler can launch multiple tasks per heartbeat, and with your code, it will launch many maps or many reduces (depending on which one it had fewer of), but not both.

The right thing would be to add a while loop outside the current for loop. On each iteration of the while loop, you'd order the task types (as {map, reduce} or {reduce, map}) and then look for a single task to launch. You then stop the while loop when neither maps nor reduces have been launched in a particular iteration. That's what I tried to describe in my post on MAPREDUCE-961, sorry if it wasn't clear.

> Alternatively schedule different types of tasks in fair share scheduler
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1198
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1198
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1198-v1.patch
>
>
> Matei has mentioned in MAPREDUCE-961 that the current scheduler will first try to launch map tasks until canLaunthTask() returns false then look for reduce tasks. This might starve reduce task. He also mention that alternatively schedule different types of tasks can solve this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.