[jira] [Commented] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901592#comment-13901592 ] 

Jason Lowe commented on MAPREDUCE-5641:
---------------------------------------

bq. I modified the permissions from 0700 to 0701.

I don't believe that will work either, since the job history and job.xml files are 0600.  So even if the history server can see it via the execute bit it won't be able to copy it.  If we allow it to copy it then it's not secure.  With those permissions, anyone with a job ID of an active job, the job's user, and the job's staging directory can obtain the job configuration (via job.xml) and job counters (via <jobid>_1.jhist).  The information needed to pull this off is trivially available, as the first two are on the front page of the RM and the latter is in public cluster configs.

> History for failed Application Masters should be made available to the Job History Server
> -----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5641
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: applicationmaster, jobhistoryserver
>    Affects Versions: 2.2.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: MAPREDUCE-5641.patch
>
>
> Currently, the JHS has no information about jobs whose AMs have failed.  This is because the History is written by the AM to the intermediate folder just before finishing, so when it fails for any reason, this information isn't copied there.  However, it is not lost as its in the AM's staging directory.  To make the History available in the JHS, all we need to do is have another mechanism to move the History from the staging directory to the intermediate directory.  The AM also writes a "Summary" file before exiting normally, which is also unavailable when the AM fails.  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)