You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2016/07/19 15:17:20 UTC
[jira] [Commented] (YARN-5401) yarn application kill does not let
mapreduce jobs show up in jobhistory
[ https://issues.apache.org/jira/browse/YARN-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384330#comment-15384330 ]
Jason Lowe commented on YARN-5401:
----------------------------------
This is effectively a duplicate of YARN-2261. MapReduce history requires the MapReduce ApplicationMaster to generate the history when it completes. hadoop job -kill or mapred job -kill accomplishes the kill by having the client connect to the MapReduce ApplicationMaster for the job and asks it to kill the job. Since this goes through the ApplicationMaster it allows the history to be generated properly.
When the kill is done via YARN then the ApplicationMaster is not involved. The ResourceManager kills the AM without the AM's knowledge. This is similar to kill vs. kill -9 (i.e.: SIGTERM vs SIGKILL) in POSIX. The former allows the application to perform cleanup tasks on the way down, while the latter mercilessly kills the process without any chance for cleanup.
Since YARN does not allow the application to specify a cleanup task to be performed when the app dies the MapReduce framework doesn't get a chance to finish generating the history for the job.
> yarn application kill does not let mapreduce jobs show up in jobhistory
> -----------------------------------------------------------------------
>
> Key: YARN-5401
> URL: https://issues.apache.org/jira/browse/YARN-5401
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Environment: centos 6.6
> apache hadoop 2.6.4
> Reporter: Nikhil Mulley
>
> Hi,
> Its been found in our cluster running apache hadoop 2.6.4, that while the mapreduce jobs that are killed with 'hadoop job -kill' command do end up have the job and its counters to jobhistory server but when 'yarn application -kill' is used on mapreduce application, job does not show up in jobhistory server interface.
> Is this intentional? If so, any particular reasons?
> It would be better to have mapreduce application history reported on jobhistory irrespective of whether kill is performed using yarn application cli or hadoop job cli.
> thanks,
> Nikhil
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org