Changing permissions of a Spark on YARN application subfolder under YARN_LOCAL_DIRS or SPARK_YARN_STAGING_DIR

[Vincenzo Eduardo Padulano <vi...@cern.ch>]

Dear all,

We're running into some permission issues with a simple setup of Spark 
on YARN.

User A starts the YARN resourcemanager on machine 1 and YARN nodemanager 
on machine 2

User B starts a Spark application (with spark.master = "yarn") on machine 1

We have already changed some parameters in HADOOP/YARN/SPARK, namely

* in yarn-site.xml:

<property>
   <name>yarn.nodemanager.local-dirs</name>
   <value>/tmp</value>
</property>

<property>
   <name> </name>
   <value>read,write,execute,delete</value>
</property>

<property>
<name>yarn.nodemanager.default-container-executor.log-dirs.permissions</name>
   <value>777</value>
</property>

* in core-default.xml:

<property>
   <name>fs.permissions.umask-mode</name>
   <value>000</value>
   <description>
     The umask used when creating files and directories.
     Can be in octal or in symbolic. Examples are:
     "022" (octal for u=rwx,g=r-x,o=r-x in symbolic),
     or "u=rwx,g=rwx,o=" (symbolic for 007 in octal).
   </description>
</property>

* in spark-defaults.conf:

spark.yarn.stagingDir /tmp/spark-yarn-staging-dir


This in turn makes folders under the YARN log directory (/tmp/userlogs) 
have permissions 777, while YARN local directory for the specific user 
(/tmp/usercache/userA/) have just 750 permissions.

Even more weirdly, when user A starts a Spark application, the current 
application directory under the YARN local directory folder, for example 
/tmp/usercache/userA/appcache/application_1613995549456_0001, has the 
following permissions:

drwx--x---. 34 userA userA 4096 Feb 22 13:31 application_1613995549456_0001


At the same time, the spark staging directory looks to have 777 
permissions on all subfolders until 
/tmp/spark-yarn-staging-dir/userA/.sparkStaging/ . But following 
subfolders after that, those that get created during an application, 
have only 700 permission!

This stops user B from sending Spark applications to the YARN cluster 
whatsoever, with errors like

File 
file:/tmp/spark-yarn-staging-dir/userB/.sparkStaging/application_1613997737582_0001/scala-library-2.12.10.jar 
does not exist

And why are certain that those jars exist. In addition, we tried to 
quickly change the permissions of the application folder to 777 on the 
fly just after it starts, that makes the application run fine without 
any errors. We have tried many parameters and we're stuck right now, we 
just think that it has to do somehow with the fact that 
yarn.nodemanager.default-container-executor.log-dirs.permissions accepts 
the classic user/group/all values whereas 
yarn.nodemanager.runtime.linux.sandbox-mode.local-dirs.permissions only 
has this comma separated list of values syntax that doesn't allow for 
extending the permissions to group/all.

We hope somebody will be able to help us out, thanks in advance.

Cheers,

Vincenzo


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org