[jira] [Updated] (YARN-11987) container-executor delete-as-user fails to rmdir private filecache directories (rmdir is only attempted as the NM user)

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-11987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated YARN-11987:
----------------------------------
    Labels: pull-request-available  (was: )

> container-executor delete-as-user fails to rmdir private filecache directories (rmdir is only attempted as the NM user)
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-11987
>                 URL: https://issues.apache.org/jira/browse/YARN-11987
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: container-executor, nodemanager
>            Reporter: YUBI LEE
>            Priority: Major
>              Labels: pull-request-available
>
> When a ContainerLocalizer exits while some resources are still in DOWNLOADING state (e.g. the container is killed during localization), LocalizerRunner cleans up by submitting a single {{FileDeletionTask}} with a null subDir and the localization dirs plus their {{_tmp}} dirs as baseDirs. With LinuxContainerExecutor this reaches container-executor as {{delete as user}} with an empty relative path, so {{delete_as_user()}} sets {{needs_tt_user = 1}} for each baseDir.
> {{delete_path()}} then deletes the directory contents as the run-as user (which succeeds), but performs the final rmdir of the top-level directory *only* as the NM user via {{rmdir_as_nm()}}:
> {code:c}
>   /*
>    * If required, do the final rmdir as root on the top level.
>    * That handles the case where the top level directory is in a directory
>    * owned by the node manager.
>    */
>   if (needs_tt_user) {
>     return rmdir_as_nm(full_path);
>   }
> {code}
> This handles the intended case — a top-level directory sitting in an NM-owned parent, e.g. {{usercache/<user>}} inside {{usercache/}} — but breaks the opposite case. Private filecache entries ({{usercache/<user>/filecache/<id>}}) sit in a directory created by ContainerLocalizer with hardcoded perms 0710 and owned by the run-as user ({{FILECACHE_PERMS}}), on which the NM user only has group {{--x}}. The rmdir fails with EACCES even though the run-as user could have removed the directory:
> {noformat}
> Shell execution returned exit code: 255.
> main : run as user is nobody
> main : requested yarn user is alice
> rmdir of /data/01/yarn/nm/usercache/alice/filecache/72407/ failed - Permission denied
> ...
> ERROR nodemanager.LinuxContainerExecutor: DeleteAsUser for
> /data/16/yarn/nm/usercache/alice/filecache/72408 ... returned with exit code: 255
> {noformat}
> So every localization aborted mid-download logs an error and leaves an empty directory behind. These are never retried (the resource is already gone from the LocalResourcesTracker/state store), so they accumulate indefinitely — inode growth and log noise. This is likely the undiagnosed cause behind YARN-9157 (same {{DeleteAsUser ... filecache ... exit code: 255}}, inode depletion) and related to the permission-model discussion in YARN-11920.
> The failure occurs in both secure and non-secure clusters: the parent filecache dir is always owned by the run-as user (the real user, or {{nonsecure-mode.local-user}} such as {{nobody}}) and is never writable by the NM user. Cache eviction ({{yarn.nodemanager.localizer.cache.target-size-mb}}) is not affected: it passes the directory as the subDir, so its final rmdir already runs as the run-as user.
> Will put up a PR that fixes this in {{delete_path()}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org