You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "magnuma3 (via GitHub)" <gi...@apache.org> on 2026/05/07 09:18:24 UTC
[PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
magnuma3 opened a new pull request, #8474:
URL: https://github.com/apache/hadoop/pull/8474
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
When a user container exits with code 22 or 24, the NodeManager becomes unhealthy and no more containers are allocated to that node. This situation can be resolved by restarting the NodeManager.
It can be reproduced immediately by running Scala Spark wordcount job that exits with code 22.
I propose to fix this by wrapping exit code 22 or 24 with different exit code, so that ConfigurationException that causes NodeManager to become unhealthy is not triggered.
```
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Obtaining the exit code...
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Docker inspect command: /usr/bin/docker inspect --format {{.State.ExitCode}} container_e161_1711009858797_8304894_01_000015
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Exit code from docker inspect: 22
2024-09-23 18:50:14,360 INFO nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(532)) - Wrote the exit code 22 to /data6/hadoop/yarn/local/nmPrivate/application_1711009858797_8304894/container_e161_1711009858797_8304894_01_000015/container_e161_1711009858797_8304894_01_000015.pid.exitcode
2024-09-23 18:50:14,381 ERROR launcher.ContainerLaunch (ContainerLaunch.java:call(340)) - Failed to launch container due to configuration error.
org.apache.hadoop.yarn.exceptions.ConfigurationException: Linux Container Executor reached unrecoverable exception
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleExitCode(LinuxContainerExecutor.java:615)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:573)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:513)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:323)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:1099)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:166)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
... 8 more
```
### How was this patch tested?
It can be reproduced immediately by running Scala Spark wordcount job that exits with code 22.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4418000005
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 17s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 17m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 57s | | trunk passed |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed |
| +1 :green_heart: | cc | 0m 41s | | the patch passed |
| +1 :green_heart: | golang | 0m 41s | | the patch passed |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 0m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 22m 43s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt) | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 88m 9s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| | TEST-cetest |
| | hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.TestFpgaDiscoverer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 76d011a2ab78 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 643ae1fdd2c0d9540b8e0284a812a4e199cfe5db |
| Default Java | Debian-25.0.3+9-2-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/testReport/ |
| Max. process+thread count | 581 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/console |
| versions | git=2.47.3 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4397123764
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed |
| +1 :green_heart: | cc | 0m 39s | | the patch passed |
| +1 :green_heart: | golang | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 22m 48s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt) | hadoop-yarn-server-nodemanager in the patch passed. |
| -1 :x: | asflicense | 0m 23s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 84m 59s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| | TEST-cetest |
| | hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.TestFpgaDiscoverer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux 60e96f3ea54a 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e0af25692c1dc3ff3e34dc04583b2b5fd10d069 |
| Default Java | Debian-25.0.3+9-2-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/testReport/ |
| Max. process+thread count | 753 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/console |
| versions | git=2.47.3 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4417646467
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 25s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 48s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 1m 48s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 31s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 0m 56s | | the patch passed |
| +1 :green_heart: | golang | 0m 56s | | the patch passed |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 0m 57s | | the patch passed |
| +1 :green_heart: | golang | 0m 57s | | the patch passed |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 23m 25s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 115m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 6e91e43aa13b 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 643ae1fdd2c0d9540b8e0284a812a4e199cfe5db |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/testReport/ |
| Max. process+thread count | 629 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/console |
| versions | git=2.43.7 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4418524681
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 1s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | cc | 0m 40s | | the patch passed |
| +1 :green_heart: | golang | 0m 40s | | the patch passed |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | cc | 0m 39s | | the patch passed |
| +1 :green_heart: | golang | 0m 39s | | the patch passed |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | spotbugs | 0m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 22m 48s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 92m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc golang |
| uname | Linux 9f2a725877d5 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 643ae1fdd2c0d9540b8e0284a812a4e199cfe5db |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/testReport/ |
| Max. process+thread count | 615 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4396560890
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 26s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 21s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 55s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK Red Hat, Inc.-21.0.11+10-LTS |
| +1 :green_heart: | cc | 0m 59s | | the patch passed |
| +1 :green_heart: | golang | 0m 59s | | the patch passed |
| +1 :green_heart: | javac | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK Red Hat, Inc.-17.0.19+10-LTS |
| +1 :green_heart: | cc | 0m 57s | | the patch passed |
| +1 :green_heart: | golang | 0m 57s | | the patch passed |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 24m 15s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not generate ASF License warnings. |
| | | 108m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux fbcdc01709b1 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e0af25692c1dc3ff3e34dc04583b2b5fd10d069 |
| Default Java | Red Hat, Inc.-17.0.19+10-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.11.0.10-1.el8_10.x86_64:Red Hat, Inc.-21.0.11+10-LTS /usr/lib/jvm/java-17-openjdk-17.0.19.0.10-1.el8_10.x86_64:Red Hat, Inc.-17.0.19+10-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/testReport/ |
| Max. process+thread count | 615 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/console |
| versions | git=2.43.7 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#discussion_r3212206555
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c:
##########
@@ -653,6 +653,16 @@ static int validate_run_as_user_commands(int argc, char **argv, int *operation)
}
}
+static int wrap_exit_code(int exit_code) {
+ if (exit_code == INVALID_CONTAINER_EXEC_PERMISSIONS || exit_code == INVALID_CONFIG_FILE) {
+ int wrap_code = WRAPPED_EXIT_CODE_USER_CONTAINER_FAILED;
+ fprintf(LOGFILE, "Wrapped exit code of user container from %d to %d to avoid NodeManager unhealthy...\n", exit_code, wrap_code);
Review Comment:
The log message "to avoid NodeManager unhealthy..." is grammatically unclear and doesn’t explain what condition is being avoided. Consider rewording to something like "to avoid NodeManager becoming unhealthy" and optionally mention it’s avoiding the ConfigurationException path in LinuxContainerExecutor for exit codes 22/24.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h:
##########
@@ -109,7 +109,8 @@ enum errorcodes {
TOO_LONG_EXECUTOR_PATH = 81,
CANNOT_GET_EXECUTABLE_NAME_FROM_KERNEL = 82,
CANNOT_GET_EXECUTABLE_NAME_FROM_PID = 83,
- WRONG_PATH_OF_EXECUTABLE = 84
+ WRONG_PATH_OF_EXECUTABLE = 84,
+ WRAPPED_EXIT_CODE_USER_CONTAINER_FAILED = 85
};
Review Comment:
New exit code 85 is added to util.h, but get_error_message() in util.c doesn’t have a case for it, so container-executor will log "Unknown error code" for this wrapped value. Also, util.h error codes appear to be mirrored by LinuxContainerExecutor.ExitCode in Java (up to 84), so consider updating those mappings to keep enums/messages consistent for diagnostics.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c:
##########
@@ -653,6 +653,16 @@ static int validate_run_as_user_commands(int argc, char **argv, int *operation)
}
}
+static int wrap_exit_code(int exit_code) {
+ if (exit_code == INVALID_CONTAINER_EXEC_PERMISSIONS || exit_code == INVALID_CONFIG_FILE) {
+ int wrap_code = WRAPPED_EXIT_CODE_USER_CONTAINER_FAILED;
+ fprintf(LOGFILE, "Wrapped exit code of user container from %d to %d to avoid NodeManager unhealthy...\n", exit_code, wrap_code);
+ return wrap_code;
+ } else {
+ return exit_code;
+ }
Review Comment:
wrap_exit_code() changes user container exit codes 22 and 24 to a single wrapped value (85). This masks the original container exit code in YARN diagnostics and makes 22 vs 24 indistinguishable; consider preserving the original information (e.g., distinct wrapped codes per original value, or an offset scheme) and/or surfacing the original exit code in diagnostics so users can still debug failures accurately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11959. NodeManager becomes unhealthy when container exits with code 22 or 24 [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8474:
URL: https://github.com/apache/hadoop/pull/8474#issuecomment-4397722636
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04 |
| +1 :green_heart: | cc | 0m 40s | | the patch passed |
| +1 :green_heart: | golang | 0m 40s | | the patch passed |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| +1 :green_heart: | cc | 0m 41s | | the patch passed |
| +1 :green_heart: | golang | 0m 41s | | the patch passed |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 22m 48s | | hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not generate ASF License warnings. |
| | | 80m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8474 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux 868c4341055f 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2e0af25692c1dc3ff3e34dc04583b2b5fd10d069 |
| Default Java | Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/testReport/ |
| Max. process+thread count | 612 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8474/1/console |
| versions | git=2.43.0 maven=3.9.11 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org