You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (JIRA)" <ji...@apache.org> on 2013/01/04 16:30:12 UTC
[jira] [Created] (MAPREDUCE-4912) Investigate ways to clean up
double job commit prevention
Robert Joseph Evans created MAPREDUCE-4912:
----------------------------------------------
Summary: Investigate ways to clean up double job commit prevention
Key: MAPREDUCE-4912
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4912
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Joseph Evans
Once MAPREDUCE-4819 goes in it fixes the issue where an OutputCommiter can double commit a job. So that the output will never be touched after the job informs externally of success or failure.
The code and design could potentially use some cleanup and refactoring.
Issues brought up that should be investigated include:
# reporting KILL for killed jobs if they crash after the kill happens instead of error.
# using the job history log for recording the commit status instead of separate external files in HDFS.
# Placing the recovery/retry logic in the commit handler instead of the MRAppMaster, and having the recovery service replay the logs as it normally does for recovery.
This is not meant to be things that must be done, but alternatives that might clean up the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira