You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "wuchang (JIRA)" <ji...@apache.org> on 2017/05/02 08:00:15 UTC
[jira] [Created] (YARN-6543) yarn application's privilege is
determined by yarn process creator instead of yarn application user.
wuchang created YARN-6543:
-----------------------------
Summary: yarn application's privilege is determined by yarn process creator instead of yarn application user.
Key: YARN-6543
URL: https://issues.apache.org/jira/browse/YARN-6543
Project: Hadoop YARN
Issue Type: Bug
Reporter: wuchang
My application is a pyspark application which is impersonated by user 'wuchang'
My application infomation is :
{code}
Application Report :
Application-Id : application_1493004858240_0007
Application-Name : livy-session-6
Application-Type : SPARK
User : wuchang
Queue : root.wuchang
Start-Time : 1493708942748
Finish-Time : 0
Progress : 10%
State : RUNNING
Final-State : UNDEFINED
Tracking-URL : http://10.120.241.82:34462
RPC Port : 0
AM Host : 10.120.241.82
Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds
Diagnostics :
{code}
And the process is :
{code}
appuser 25454 25872 0 15:09 ? 00:00:00 bash /data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/default_container_executor.sh
appuser 25456 25454 0 15:09 ? 00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp '-Dspark.ui.port=0' '-Dspark.driver.port=40969' -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar 1> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stdout 2> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stderr
appuser 25468 25456 2 15:09 ? 00:00:09 /home/jdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp -Dspark.ui.port=0 -Dspark.driver.port=40969 -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004 -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
appuser 26936 25846 0 15:16 pts/0 00:00:00 grep --color=auto application_1493004858240_0007
{code}
The main problem is that the application user is "wuchang" , but the yarn application is created by my OS super-user "appuser" , so , the privilege becomes the problem. My code always run as the privilege of appuser instead of "wuchang".
For example , below is the pyspark code:
{code}
import os
os.system("hadoop fs -rm -r /user/appuser/test.dat")
{code}
user "wuchang" should not have privilege to remove the file test.dat which located in the home directory of appuser. But since the yarn application process is created by "appuser", it does, although the yarn application user is "wuchang".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org