[jira] Commented: (HADOOP-4981) Prior code fix in Capacity Scheduler prevents speculative execution in jobs

["Vivek Ratan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660684#action_12660684 ] 

Vivek Ratan commented on HADOOP-4981:
-------------------------------------

The intent of the original fix was to solve the following problem: if  job has high memory requirements, you want to return no task to the TT. However, you want to do this only if the job has tasks to run (otherwise, you underutilize the TTs). Thsi problem still needs to be solved, but not in the way it was done earlier. Not having any pending tasks does not mean that a job does not have any task to run. 

> Prior code fix in Capacity Scheduler prevents speculative execution in jobs
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4981
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4981
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Vivek Ratan
>            Priority: Critical
>
> As part of the code fix for HADOOP-4035, the Capacity Scheduler obtains a task from JobInProgress (calling obtainNewMapTask() or obtainNewReduceTask()) only if the number of pending tasks for a job is greater than zero (see the if-block in TaskSchedulingMgr.getTaskFromJob()). So, if a job has no pending tasks and only has running tasks, it will never be given a slot, and will never have a chance to run a speculative task. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.