[jira] [Updated] (MAPREDUCE-5848) MapReduce counts forcibly preempted containers as FAILED

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated MAPREDUCE-5848:
----------------------------------

     Target Version/s: 2.5.0
    Affects Version/s: 2.1.0-beta

Moving this to MAPREDUCE since it's an issue with the MaoReduce AM and not YARN.

Note that the small patch may not be sufficient to completely solve the issue of preempted tasks being interpreted as FAILED rather than KILLED.  Due to races between the AM heartbeat to the RM and the AM receiving task status updates via the task umbilical the AM may still think that a task attempt failed even though it was preempted.  When a task attempt receives SIGTERM from the NM it causes the FileSystem to close via the shutdown hook and often causes exceptions within the task.  Those exceptions are reported as task failure via the task umbilical.  If that status arrives at the AM before the AM receives the preempted container status from the RM then the AM counts it as a task failure rather than a task preempt.

> MapReduce counts forcibly preempted containers as FAILED
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-5848
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5848
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.1.0-beta
>            Reporter: Carlo Curino
>            Assignee: Subramaniam Krishnan
>         Attachments: YARN-1958.patch
>
>
> The MapReduce AM is considering a forcibly preempted container as FAILED, while I think it should be considered as KILLED (i.e., not count against the maximum number of failures). 



--
This message was sent by Atlassian JIRA
(v6.2#6252)