non-HDFS yarn permission issue

[Gurmeet Singh <gu...@robinsystems.com>]

Hi,
   I am trying to test a non-HDFS distributed filesystem (ceph) with yarn and I think I am running into permissions issue. I am trying to run the DistributedShell example and it fails.

hdfs@hadoop25:~$ yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.3.0.jar -shell_command uptime
14/10/01 17:21:11 INFO distributedshell.Client: Initializing Client
14/10/01 17:21:11 INFO distributedshell.Client: Running Client
...
14/10/01 17:21:27 INFO distributedshell.Client: Application did not finish. YarnState=FAILED, DSFinalStatus=FAILED. Breaking monitoring loop
14/10/01 17:21:27 ERROR distributedshell.Client: Application failed to complete successfully


Looking at the nodemanager logs, I see that localization failed for

2014-10-01 17:21:26,460 DEBUG org.apache.hadoop.fs.ceph.CephFileSystem: getFileStatus ceph://10.7.202.1:6789/user/hdfs/DistributedShell/1/shellCommands
2014-10-01 17:21:26,464 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: Permission denied
2014-10-01 17:21:26,500 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { ceph://10.7.202.1:6789/user/hdfs/DistributedShell/1/shellCommands<http://10.7.202.1:6789/user/hdfs/DistributedShell/1/shellCommands>, 1412184072540, FILE, null }, Permission denied
2014-10-01 17:21:26,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource ceph://10.7.202.1:6789/user/hdfs/DistributedShell/1/shellCommands<http://10.7.202.1:6789/user/hdfs/DistributedShell/1/shellCommands> transitioned from DOWNLOADING to FAILED

I see that the file does actually exist

hdfs@hadoop25:~$ hdfs dfs -cat /user/hdfs/DistributedShell/1/shellCommands
uptime

Also, if I submit the job as the yarn user, then it completes successfully. Does it mean that since the Nodemanager is running as the yarn user, it can only localize files that if has read permissions on. But then how can I make this work in a multiuser environment and also this works fine with HDFS, so how does it work fine with HDFS.

thanks

Gurmeet