[jira] [Resolved] (MAPREDUCE-262) Optimize finding of speculative tasks

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved MAPREDUCE-262.
----------------------------------------

    Resolution: Fixed

Spec exec got optimized somewhere along the way. Closing.

> Optimize finding of speculative tasks
> -------------------------------------
>
>                 Key: MAPREDUCE-262
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-262
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Amar Kamat
>
> Assuming HADOOP-2119 provides better data structures for handling running TIPs, finding new speculative tasks can be further optimized. Two of which could be 
> 1) {{conf.getMapSpeculativeExecution()}} and {{conf.getReduceSpeculativeExecution()}} should be moved to {{JobInProgress}}. A simple check for this boolean can prove useful before checking for speculative tasks. This will be useful for jobs with large maps and reducers where scanning all the TIPs  can be costly. 
> 2) Since the progress of a TIP changes only when {{TaskInProgress.recomputeProgress()}} is invoked, it makes more sense to check for speculation in {{JobInProgress.updateTaskStatus()}} and move the TIPs that can be speculated to the front of the running queue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)