You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Aidi Pi (JIRA)" <ji...@apache.org> on 2017/08/09 16:22:00 UTC
[jira] [Created] (YARN-6976) Some containers take a long time in
KILLING state after the application is finished.
Aidi Pi created YARN-6976:
-----------------------------
Summary: Some containers take a long time in KILLING state after the application is finished.
Key: YARN-6976
URL: https://issues.apache.org/jira/browse/YARN-6976
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager, resourcemanager
Affects Versions: 2.7.3
Environment: OS: Ubuntu 16.04, Java: JDK1.8, Docker: seqenceid/hadoop-2.4.0
Reporter: Aidi Pi
I use Docker as the container of YARN and ran Spark applications. In some runs, the resource manager log indicates that the application is done. However, some nodemanager logs indicates that the containers on this node are still in RUNNING state then enter KILLING state. They spend a long time (about 20s) in KILLING state before terminated.
In this case, 3 containers were still running after the app entered FINISHED state.
Below is the tail of RM and NM logs:
{panel:title=RM log}
2017-08-08 15:11:34,009 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1502226348464_0002 State change from FINISHING to FINISHED
{panel}
{panel:title=NM log}
2017-08-08 15:11:51,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000003 transitioned from KILLING to EXITED_WITH_SUCCESS
2017-08-08 15:11:51,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000010 transitioned from KILLING to EXITED_WITH_SUCCESS
2017-08-08 15:11:51,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000016 transitioned from KILLING to EXITED_WITH_FAILURE
2017-08-08 15:11:51,309 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=eddie OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1502226348464_0002 CONTAINERID=container_1502226348464_0002_01_000003
2017-08-08 15:11:51,351 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000003 transitioned from EXITED_WITH_SUCCESS to DONE
2017-08-08 15:11:51,351 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=eddie OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1502226348464_0002 CONTAINERID=container_1502226348464_0002_01_000010
2017-08-08 15:11:51,351 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000010 transitioned from EXITED_WITH_SUCCESS to DONE
2017-08-08 15:11:51,357 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=eddie OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1502226348464_0002 CONTAINERID=container_1502226348464_0002_01_000016
2017-08-08 15:11:51,357 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1502226348464_0002_01_000016 transitioned from EXITED_WITH_FAILURE to DONE
{panel}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org