You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2012/10/23 22:59:13 UTC
[jira] [Commented] (YARN-167) AM stuck in KILL_WAIT for days when
node is lost in the middle
[ https://issues.apache.org/jira/browse/YARN-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482682#comment-13482682 ]
Ravi Prakash commented on YARN-167:
-----------------------------------
This is my best guess for what is happening. Imagine a job is running, and we send it the KILL signal. The job transitions from RUNNING to KILL_WAIT. The task transitions from RUNNING to KILL_WAIT. However, some task attempts may be in COMMIT_PENDING state. Pasting state graph here for reference
!TaskAttemptStateGraph.jpg!
If the TA_DONE event is queued in the event queue before the TA_KILL event, then the task attempt is transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP (which we would think should've transitioned to KILL_CONTAINER_CLEANUP, because hey, we sent it TA_KILL and it was in COMMIT_PENDING). Afterwards, the Task Attempt transitions from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED. In either of these states TA_KILL is ignored. So the Task stays in KILL_WAIT and consequently the Job too.
> AM stuck in KILL_WAIT for days when node is lost in the middle
> --------------------------------------------------------------
>
> Key: YARN-167
> URL: https://issues.apache.org/jira/browse/YARN-167
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 0.23.3
> Reporter: Ravi Prakash
> Assignee: Vinod Kumar Vavilapalli
> Attachments: TaskAttemptStateGraph.jpg
>
>
> We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a few maps running. All these maps were scheduled on nodes which are now in the RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira