You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "KWON BYUNGCHANG (Jira)" <ji...@apache.org> on 2026/05/06 09:06:00 UTC
[jira] [Created] (YARN-11959) NodeManager becomes unhealthy when container exits with code 22 or 24
KWON BYUNGCHANG created YARN-11959:
--------------------------------------
Summary: NodeManager becomes unhealthy when container exits with code 22 or 24
Key: YARN-11959
URL: https://issues.apache.org/jira/browse/YARN-11959
Project: Hadoop YARN
Issue Type: Bug
Reporter: KWON BYUNGCHANG
When a user container exits with code 22 or 24, the NodeManager becomes unhealthy and no more containers are allocated to that node. This situation can be resolved by restarting the NodeManager.
It can be reproduced immediately by running Scala Spark wordcount job that exits with code 22.
I propose to fix this by wrapping exit code 22 or 24 with different exit code, so that ConfigurationException that causes NodeManager to become unhealthy is not triggered.
{noformat}
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Obtaining the exit code...
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_e161_1711009858797_8304894_01_000015
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exit code from docker inspect: 22
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Wrote the exit code 22 to /data6/hadoop/yarn/local/nmPrivate/application_1711009858797_8304894/container_e161_1711009858797_8304894_01_000015/container_e161_1711009858797_8304894_01_000015.pid.exitcode
2024-09-23 18:50:14,381 ERROR launcher.ContainerLaunch (ContainerLaunch.java:call(340)) - Failed to launch container due to configuration error.
org.apache.hadoop.yarn.exceptions.ConfigurationException: Linux Container Executor reached unrecoverable exception
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleExitCode(LinuxContainerExecutor.java:615)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:573)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:513)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:323)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:1099)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:166)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
... 8 more {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org