[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory

["Ming Ma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15256607#comment-15256607 ] 

Ming Ma commented on MAPREDUCE-6315:
------------------------------------

Thanks [~jira.shegalov]! We definitely need to support this scenario. There was some discussion about providing clean-up functionality in YARN-2261 and MAPREDUCE-4428 so that history files can be moved to final location properly. But it isn't clear when we plan to provide such functionality at YARN layer and it seems like a larger effort. The patch here is more targeted and can take care of the issue we have had, at least until the end-to-end clean-up functionality is available. What do you think? Specific comments for the jira:

* This will enable "mapred job -logs" usage. How about the "jobhistory URL http://...../jobhistory/job/job_ returns job not found" scenario, is it easy to add the redirect at that level?
* globStatus might return empty list. So it might be better to change from {{if (jhStats != null) }} to something like {{if (jhStats != null && jhStats.length > 0)}}.
* Is the output format change in JobHistoryParser required? Wonder if there is any backward compatibility issue if some tools have assumption about this.

> Implement retrieval of logs for crashed MR-AM via jhist in the staging directory
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6315
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: client, mr-am
>    Affects Versions: 2.7.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>            Priority: Critical
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, MAPREDUCE-6315.003.patch
>
>
> When all AM attempts crash, there is no record of them in JHS. Thus no easy way to get the logs. This JIRA automates the procedure by utilizing the jhist file in the staging directory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)