[jira] [Comment Edited] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909246#comment-13909246 ] 

Vinod Kumar Vavilapalli edited comment on MAPREDUCE-5641 at 2/22/14 5:16 AM:
-----------------------------------------------------------------------------

Unless I am missing something, I still don't understand why my proposal [here|https://issues.apache.org/jira/browse/MAPREDUCE-5641?focusedCommentId=13906448&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13906448] of making JHS talk to RM about the application-information is not enough to begin with. It can be extended in future to talk to AHS to obtain more information.

To your question about scale, Jason did answer that it can be done on demand for only those apps which don't have history files.


was (Author: vinodkv):
I still don't understand why my proposal [here|https://issues.apache.org/jira/browse/MAPREDUCE-5641?focusedCommentId=13906448&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13906448] of making JHS talking to RM about the application-information is not enough to begin with. It can be extended in future to talk to AHS. To your question about scale, Jason did answer that it can be done on demand for only those apps which don't have history files.

> History for failed Application Masters should be made available to the Job History Server
> -----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5641
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, jobhistoryserver
>    Affects Versions: 2.2.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-5641.patch, MAPREDUCE-5641.patch
>
>
> Currently, the JHS has no information about jobs whose AMs have failed.  This is because the History is written by the AM to the intermediate folder just before finishing, so when it fails for any reason, this information isn't copied there.  However, it is not lost as its in the AM's staging directory.  To make the History available in the JHS, all we need to do is have another mechanism to move the History from the staging directory to the intermediate directory.  The AM also writes a "Summary" file before exiting normally, which is also unavailable when the AM fails.  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)