[jira] Updated: (MAPREDUCE-517) The capacity-scheduler should assign multiple tasks per heartbeat

["Arun C Murthy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated MAPREDUCE-517:
------------------------------------

    Attachment: MAPREDUCE-517_yhadoop20.patch

Updated patch for y20.

Highlights:

# CS assigns multiple tasks per heartbeat, at most 1 off-switch task per heartbeat.
# I've incorporated HADOOP-538 also to ensure jobs at the head of the queue do not aggressively grab tasks hurting locality for others.
## The implementation tracks 'number of scheduling opportunities' missed by a job and gets jobs to use that to prevent starvation.
## I've also added 'pace' to the back-off by getting ensuring jobs do not back-off as aggressively as they make progress.
## The patch also gets small jobs to back off less vis-a-vis larger jobs by ensuring the backoff considers the #maps in the jobs.
## The patch also ensures jobs with no locality e.g. sleep-job/randomwriter do not care about backoff since it doesn't make sense at all.

> The capacity-scheduler should assign multiple tasks per heartbeat
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-517
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-517
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>         Attachments: HADOOP-5090-20090504.txt, HADOOP-5090-20090506.txt, HADOOP-5090-20090604.txt, MAPREDUCE-517_yhaddop20.patch, MAPREDUCE-517_yhadoop20.patch
>
>
> HADOOP-3136 changed the default o.a.h.mapred.JobQueueTaskScheduler to assign multiple tasks per TaskTracker heartbeat, the capacity-scheduler should do the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.