[jira] Commented: (HADOOP-2141) speculative execution start up condition based on completion time

["Amar Kamat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574848#action_12574848 ] 

Amar Kamat commented on HADOOP-2141:
------------------------------------

Consider a scenario where the reducer does some kind of _fancy_, _hi-fi_ and _time consuming_ work based on some events. One such example would be to do some kind of learning for specific words. So in some cases the reducers will finish off faster (the main logic is never triggered) while in some case it might take a lot longer. So comparing the speed across reducers might not be good idea in this case. Also taking the progress rate might not be a good idea since sometimes the reducer just scans and sometimes it does some fancy computing. Hence in such cases speculation wont help. So my question is
-  should we consider the above mentioned scenario ( and similar others) into account (if possible) or is it too much to consider
-  how much is the performance overhead ( across jobs/cluster usage/utility) in case of speculations

> speculative execution start up condition based on completion time
> -----------------------------------------------------------------
>
>                 Key: HADOOP-2141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2141
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.15.0
>            Reporter: Koji Noguchi
>            Assignee: Arun C Murthy
>             Fix For: 0.17.0
>
>
> We had one job with speculative execution hang.
> 4 reduce tasks were stuck with 95% completion because of a bad disk. 
> Devaraj pointed out 
> bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.
> It would be nice if speculative execution also starts up when tasks stop making progress.
> Devaraj suggested 
> bq. Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.