[jira] Updated: (HADOOP-76) Implement speculative re-execution of reduces

["Sanjay Dahiya (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/HADOOP-76?page=all ]

Sanjay Dahiya updated HADOOP-76:
--------------------------------

    Attachment: Hadoop-76_1.patch

Here is update patch for review

changes in this patch 
 - Moved temp files to (mapred.system.dir)/<jobid> 
-  2 new variables in TIP - 
        -    runningTasks ( to track currently running instances of attempts for same task, earlier this was done using a boolean) 
        -    hasCompletedTask ( for reduces if a attempts succeeds then all subsequent failures for same TIP should be ignored, existing setup tries to schedule another task for that.) 
- Added a phaseFileSystem, which takes a jobid, tipid, taskid and creates all files in (mapred.system.dir)/<jobid>/<tipid>/<taskid>, these are moved to their final location on commit or deleted. 
- Task Constructor needs tip id now, its passed in RPC as well. 
- Task.localizeConfiguration adds tip id to conf. 
- Minor change in FSDirectory to add exception message in log if rename fails. 

+Planning to add an example for using PhasedFileSystem - changing RandomWriter to use PhasedFileSystem, as Devaraj suggested in a separate patch on the same issue. 

thanks owen for review. 

> Implement speculative re-execution of reduces
> ---------------------------------------------
>
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, Hadoop-76_1.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are renamed when the task completes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira