[jira] Commented: (MAPREDUCE-2205) FairScheduler should not re-schedule jobs that have just been preempted

["Scott Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971832#action_12971832 ] 

Scott Chen commented on MAPREDUCE-2205:
---------------------------------------

Joydeep: Thanks. I will submit a patch based on our discussion yesterday.

> FairScheduler should not re-schedule jobs that have just been preempted
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2205
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2205
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>            Reporter: Joydeep Sen Sarma
>            Assignee: Scott Chen
>
> We have hit a problem with the preemption implementation in the FairScheduler where the following happens:
> # job X runs short of fair share or min share and requests/causes N tasks to be preempted
> # when slots are then scheduled - tasks from some other job are actually scheduled
> # after preemption_interval has passed, job X finds it's still underscheduled and requests preemption. goto 1.
> This has caused widespread preemption of tasks and the cluster going from high utilization to low utilization in a few minutes.
> After doing some analysis of the logs - one of the biggest contributing factors seems to be the scheduling of jobs when a heartbeat with multiple slots is advertised. currently it goes over all the jobs/pools (in sorted) order until all the slots are exhausted. this leads to lower priority jobs also getting scheduled (that may have just been preempted).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.