You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/03/21 03:30:00 UTC
[jira] [Commented] (YARN-11920) linux-container-executor requires flexible directory permissions.
[ https://issues.apache.org/jira/browse/YARN-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18067215#comment-18067215 ]
ASF GitHub Bot commented on YARN-11920:
---------------------------------------
edwardcapriolo commented on PR #8184:
URL: https://github.com/apache/hadoop/pull/8184#issuecomment-4102046918
@cnauroth I dont know how you say you can not reproduce this?
1. I have published exact instructions including a docker image that shows this.
2. The output is in detail in the ticket. The directory owership does not allow the app cache directory to be created
3. There are more danling pointers in the code (im fixing serious leeks in what is supposed to be secure code)
> linux-container-executor requires flexible directory permissions.
> -----------------------------------------------------------------
>
> Key: YARN-11920
> URL: https://issues.apache.org/jira/browse/YARN-11920
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Edward Capriolo
> Assignee: Edward Capriolo
> Priority: Major
> Labels: pull-request-available
>
> When running yarn node-manager directories are created which the node manager can not write to. I discussed this on the mailing list:
> {quote}Hello. I am trying to run linux-container-executor in a setup without kerberos. I want to see it "change user" and run a map reduce job.
> I have a fork of linux-container-executor with some gratuitous println:
> main : command provided 0
> 2026-01-12T19:51:25.467740715Z main : run as user is auser
> 2026-01-12T19:51:25.467750476Z main : requested yarn user is auser
> 2026-01-12T19:51:25.467760225Z main : validate_container_id
> 2026-01-12T19:51:25.467771148Z main : huh
> 2026-01-12T19:51:25.467784131Z validated command: INITIALIZE_CONTAINER
> 2026-01-12T19:51:25.467795274Z init : set_user
> 2026-01-12T19:51:25.467805332Z maybe free_user
> 2026-01-12T19:51:25.467815142Z going to check user
> 2026-01-12T19:51:25.467824798Z min id
> 2026-01-12T19:51:25.467833618Z min id 1000
> 2026-01-12T19:51:25.467842685Z Get user info
> 2026-01-12T19:51:25.467851066Z init : set_user done
> 2026-01-12T19:51:25.467860879Z initialize_app(
> 2026-01-12T19:51:25.467871118Z create user dirs
> 2026-01-12T19:51:25.467881131Z initialize_user.
> 2026-01-12T19:51:25.467890384Z created
> 2026-01-12T19:51:25.467900435Z create_log_dirs().
> 2026-01-12T19:51:25.467911090Z create container log
> 2026-01-12T19:51:25.467920790Z create_container_log_dirs
> 2026-01-12T19:51:25.467931683Z open_file_as_nm.
> 2026-01-12T19:51:25.467941717Z change_user
> 2026-01-12T19:51:25.467952667Z change_user.
> *2026-01-12T19:51:25.467962032Z Can't create directory /yarn-root/nm-local-dir/usercache/auser/appcache - Permission denied*
> 2026-01-12T19:51:25.467973350Z Did not create any app directories
> I am creating users like this:
> RUN addgroup -S hadoop
> RUN addgroup -S hdfs && adduser -S -G hdfs -H -D hdfs
> RUN addgroup -S yarn && adduser -S -G yarn -H -D yarn
> RUN addgroup yarn hadoop
> RUN addgroup -S auser && adduser -S -G auser -H -D auser
> I am launching a wordcount as "auser" like so:
> [https://github.com/edwardcapriolo/edgy-ansible/blob/main/imaging/hadoop/compositions/ha_rm_zk_pki_tls/enter_auser.sh]
> This is what teh directory inside the node manager looks like:
> nm1:/yarn-root/nm-local-dir/usercache# rm -rf auser/
> nm1:/yarn-root/nm-local-dir/usercache# ld -lahd /yarn-root/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lahd /yarn-root/
> drwxr-xr-x 1 yarn root 24 Jan 12 19:32 /yarn-root/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lahd /yarn-root/nm-local-dir/
> drwxr-xr-x 1 yarn hadoop 54 Jan 12 19:32 /yarn-root/nm-local-dir/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lahd /yarn-root/nm-local-dir/
> filecache/ nmPrivate/ usercache/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lahd /yarn-root/nm-local-dir/usercache/
> drwxr-sr-x 1 yarn hadoop 10 Jan 12 20:38 /yarn-root/nm-local-dir/usercache/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lahd /yarn-root/nm-local-dir/usercache/auser/
> drwxr-s--- 1 auser hadoop 0 Jan 12 20:38 /yarn-root/nm-local-dir/usercache/auser/
> My node manager is running as yarn
> nm1:/$ ps -ef | grep yarn
> 1 yarn 0:20 /usr/bin/java -Dproc_nodemanager
> nm1:/$ id -u yarn
> 101
> nm1:/$ id -g yarn
> 103
> nm1:/$ id -G yarn
> 103 101
> nm1:/$ id -G yarn -n
> yarn hadoop
> nm1:/$ umask
> 0022
> I am guessing that the issue is
> drwxr-s--- 1 auser hadoop 0 Jan 12 20:38 auser
> Ths directory gets owned by auser/hadoop but the group write is off?
> My yarn config is here:
> [https://github.com/edwardcapriolo/edgy-ansible/blob/main/imaging/hadoop/compositions/ha_rm_zk_pki_tls/hd_conf/yarn-site.xml#L126]
> Also manually changing it it just gets put back
> nm1:/yarn-root/nm-local-dir/usercache# chmod g+w auser/
> nm1:/yarn-root/nm-local-dir/usercache# ls -lah
> total 0
> drwxr-sr-x 1 yarn hadoop 10 Jan 12 20:38 .
> drwxr-xr-x 1 yarn hadoop 54 Jan 12 19:32 ..
> drwxrws--- 1 auser hadoop 0 Jan 12 20:38 auser
> nm1:/yarn-root/nm-local-dir/usercache# ls -lah
> total 0
> drwxr-sr-x 1 yarn hadoop 10 Jan 12 20:38 .
> drwxr-xr-x 1 yarn hadoop 54 Jan 12 19:32 ..
> drwxr-s--- 1 auser hadoop 0 Jan 12 20:38 auser
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org