You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "eubnara (via GitHub)" <gi...@apache.org> on 2026/08/22 11:22:15 UTC
[PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
eubnara opened a new pull request, #8700:
URL: https://github.com/apache/hadoop/pull/8700
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
See YARN-11987 for the full analysis. Summary:
When LocalizerRunner cleans up after an aborted localization, it batches the leftover dirs as baseDirs with an empty relative path, so `delete_as_user()` sets `needs_tt_user = 1` and `delete_path()` performs the final top-level rmdir *only* as the NM user. That works for NM-owned parents (`usercache/<user>`), but always fails with EACCES for private filecache entries (`usercache/<user>/filecache/<id>`), whose parent is owned by the run-as user with mode 0710. Every aborted localization logs a `DeleteAsUser ... exit code: 255` error and leaks an empty directory that is never retried.
This PR makes `delete_path()` try the final rmdir as the current (run-as) user first, falling back to `rmdir_as_nm()` on EACCES/EPERM when `needs_tt_user` is set:
* NM-owned parents keep working exactly as before (rmdir as user fails, NM fallback succeeds).
* User-owned parents are fixed (rmdir as user succeeds immediately).
* No privilege escalation: the NM-user rmdir already existed; the fix only tries the less-privileged user first. The fallback is restricted to EACCES/EPERM because other errors (ENOTEMPTY, EBUSY, ...) would fail identically for the NM user.
An alternative would be to fix the caller (one FileDeletionTask per path, with the path as subDir), but batching several absolute paths into one container-executor invocation is only possible through baseDirs, so that costs one exec per path and leaves the same trap for any other caller. Fixing `delete_path()` covers all callers in one place.
### How was this patch tested?
Added `test_delete_dir_in_user_owned_parent` to test-container-executor.c. The failure mode needs the final rmdir to run as a non-root NM user different from the directory owner, so the test only exercises the bug in the root invocation mode with two distinct users:
```bash
# build native, then as root:
./test-container-executor <nm-user> <run-as-user> # e.g. yarn nobody
```
Before the fix this fails the new test (rmdir_as_nm gets EACCES); after the fix it passes. Notably, without the fix even the *existing* `test_delete_user` fails in this two-user root mode — its empty-subdir scenario deletes a dir under the run-as user's `appcache`, which hits the same EACCES:
```
Testing delete_user
rmdir of .../local-1/usercache/nobody/appcache/app_3/ failed - Permission denied
FAIL: directory not deleted
```
Non-root runs (cetest in CI) skip the privileged part gracefully, same as the existing test_delete_user.
Also verified manually with the exact NM invocation against a directory laid out like a private filecache entry:
```bash
install -d -o nobody -g nobody -m 755 /data/01/yarn/nm/usercache/alice/filecache/999999
sudo -u yarn $HADOOP_YARN_HOME/bin/container-executor nobody alice 3 "" /data/01/yarn/nm/usercache/alice/filecache/999999
```
Before: exit 255, `rmdir of ... failed - Permission denied`, empty dir remains. After: exit 0, dir removed. The preserved NM-owned-parent case (`... 3 "" /data/01/yarn/nm/usercache/alice`) still succeeds.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
Generative AI: Contains content generated by Claude Code (Anthropic
Claude). The change was human-reviewed and verified on a production
cluster, and complies with the ASF Generative Tooling Guidance
(https://www.apache.org/legal/generative-tooling.html).
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8700:
URL: https://github.com/apache/hadoop/pull/8700#issuecomment-5382308598
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | mvnsite | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 85m 53s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 86m 28s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | cc | 1m 7s | | the patch passed |
| +1 :green_heart: | golang | 1m 7s | | the patch passed |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | cc | 1m 9s | | the patch passed |
| +1 :green_heart: | golang | 1m 9s | | the patch passed |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 26m 20s | | hadoop-yarn-server-nodemanager in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 153m 14s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux af70f66a0ec1 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2add9630210752f88ceb1bb74eb65e37bf41da8e |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/testReport/ |
| Max. process+thread count | 623 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8700:
URL: https://github.com/apache/hadoop/pull/8700#issuecomment-5380930392
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 18s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 52m 59s | | trunk passed |
| +1 :green_heart: | compile | 2m 54s | | trunk passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | compile | 2m 55s | | trunk passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | mvnsite | 2m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 99m 35s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 34s | | the patch passed with JDK Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | cc | 1m 34s | | the patch passed |
| +1 :green_heart: | golang | 1m 34s | | the patch passed |
| +1 :green_heart: | javac | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 36s | | the patch passed with JDK Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | cc | 1m 36s | | the patch passed |
| +1 :green_heart: | golang | 1m 36s | | the patch passed |
| +1 :green_heart: | javac | 1m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 27m 16s | | hadoop-yarn-server-nodemanager in the patch passed. |
| -1 :x: | asflicense | 0m 58s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 190m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8700 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux e042963c6bdf 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dde012b588e512a7afa8dfc1dde3a969cd9229a1 |
| Default Java | Red Hat, Inc.-17.0.20.1+1-LTS |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-21.0.12.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-21.0.12.1+1-LTS /usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64:Red Hat, Inc.-17.0.20.1+1-LTS |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/testReport/ |
| Max. process+thread count | 675 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/console |
| versions | git=2.43.7 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #8700:
URL: https://github.com/apache/hadoop/pull/8700#issuecomment-5537172824
thanks, committed to trunk/branch-3.5/branch-3.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 merged PR #8700:
URL: https://github.com/apache/hadoop/pull/8700
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8700:
URL: https://github.com/apache/hadoop/pull/8700#issuecomment-5381655007
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 84m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed |
| +1 :green_heart: | cc | 1m 10s | | the patch passed |
| +1 :green_heart: | golang | 1m 10s | | the patch passed |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 26m 21s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt) | hadoop-yarn-server-nodemanager in the patch passed. |
| -1 :x: | asflicense | 0m 34s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 168m 36s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.TestFpgaDiscoverer |
| | TEST-cetest |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8700 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang |
| uname | Linux 0ea10336079c 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dde012b588e512a7afa8dfc1dde3a969cd9229a1 |
| Default Java | Debian-25.0.4+7-1-deb13u1-Debian |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/testReport/ |
| Max. process+thread count | 575 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8700/1/console |
| versions | git=2.47.3 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11987. container-executor delete-as-user fails to rmdir private filecache directories. [hadoop]
Posted by "eubnara (via GitHub)" <gi...@apache.org>.
eubnara commented on PR #8700:
URL: https://github.com/apache/hadoop/pull/8700#issuecomment-5433177724
@cnauroth @slfan1989 Could you take a look when you have a chance?
This fixes a "Permission denied" on the final rmdir in container-executor's delete-as-user path: private filecache directories are created with 0710 owned by the run-as user, but the rmdir runs as the NM user, so empty directories pile up (see YARN-9157 for the inode-exhaustion symptom and YARN-11920 for the permission-model discussion).
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org