[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery

["Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119935#comment-13119935 ] 

Arun C Murthy commented on MAPREDUCE-2702:
------------------------------------------

Sharad, apologies for not getting around to this, but I finally spent sufficient time on this to do this important patch justice.

Overall, it looks great! Some minor nits:

# How about uniting isRecoverySupported and recoverTask into a single api: 'boolean recoverTask'. So, an OutputCommitter can return false from 'recoverTask' to indicate it doesn't support it.
# I'm a little concerned about introducing the notion of 'appAttemptId' as an integer to recoverTask. This is probably one of the very few, if not the only place, where YARN leaks into 'classic' MapReduce apis - particularly since these are user-facing apis. Maybe we can just pull it out of TaskAttemptContext.configuration?
# Other minor, cosmetic suggestions for function names:
## getJobTempDir -> getJobTempDirName
## getTaskTempDir -> getTaskTempDirName
## moveOutputs -> moveJobOutputs

Thoughts?
                
> [MR-279] OutputCommitter changes for MR Application Master recovery
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2702
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>            Priority: Blocker
>         Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch
>
>
> In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira