You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2014/07/24 11:11:06 UTC

issue about run MR job use system user

hi,maillist:
          i create a system user on a box of my hadoop cluster ,but when i
run MR job use this user ,it get a problem, the /data directory is for
mapreduce history server option, and i also add the user into hadoop group
,since the /data privilege is 775 ,so it can write by user in hadoop
group,why still cause permssion error? anyone can help?

 # useradd alex

<configuration>
        <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
        </property>
        <property>
         <name>mapreduce.jobhistory.address</name>
         <value>192.168.10.49:10020</value>
        </property>
        <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>192.168.10.49:19888</value>
        </property>
        <property>
         <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/data</value>
        </property>
.........
<configuration>
$ hadoop fs -ls /
Found 6 items
drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data

$ hadoop fs -ls /data
Found 3 items
drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe

[alex@hz23 ~]$ id
uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)


[alex@hz23 ~]$  hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
100
Number of Maps  = 2
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/07/24 17:06:23 WARN security.UserGroupInformation:
PriviledgedActionException as:alex (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
        at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
        at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
        at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
        at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
        at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
org.apache.hadoop.security.AccessControlException: Permission denied:
user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x

Re: issue about run MR job use system user

Posted by ch huang <ju...@gmail.com>.
i reslove this by make alex directory under staging directory and set the
owner to alex

On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

>  Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
>  ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by ch huang <ju...@gmail.com>.
i reslove this by make alex directory under staging directory and set the
owner to alex

On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

>  Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
>  ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by ch huang <ju...@gmail.com>.
i reslove this by make alex directory under staging directory and set the
owner to alex

On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

>  Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
>  ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by ch huang <ju...@gmail.com>.
i reslove this by make alex directory under staging directory and set the
owner to alex

On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

>  Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
>  ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by Stanley Shi <ss...@gopivotal.com>.
The "alex" should belongs to "hadoop" group on namenode;

Regards,
*Stanley Shi,*



On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

> Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
> ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by Stanley Shi <ss...@gopivotal.com>.
The "alex" should belongs to "hadoop" group on namenode;

Regards,
*Stanley Shi,*



On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

> Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
> ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by Stanley Shi <ss...@gopivotal.com>.
The "alex" should belongs to "hadoop" group on namenode;

Regards,
*Stanley Shi,*



On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

> Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
> ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

Re: issue about run MR job use system user

Posted by Stanley Shi <ss...@gopivotal.com>.
The "alex" should belongs to "hadoop" group on namenode;

Regards,
*Stanley Shi,*



On Thu, Jul 24, 2014 at 10:11 PM, java8964 <ja...@hotmail.com> wrote:

> Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be
> confirmed on the namenode?
>
> Yong
>
> ------------------------------
> Date: Thu, 24 Jul 2014 17:11:06 +0800
> Subject: issue about run MR job use system user
> From: justlooks@gmail.com
> To: user@hadoop.apache.org
>
>
> hi,maillist:
>           i create a system user on a box of my hadoop cluster ,but when i
> run MR job use this user ,it get a problem, the /data directory is for
> mapreduce history server option, and i also add the user into hadoop group
> ,since the /data privilege is 775 ,so it can write by user in hadoop
> group,why still cause permssion error? anyone can help?
>
>  # useradd alex
>
> <configuration>
>         <property>
>          <name>mapreduce.framework.name</name>
>          <value>yarn</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.address</name>
>          <value>192.168.10.49:10020</value>
>         </property>
>         <property>
>          <name>mapreduce.jobhistory.webapp.address</name>
>          <value>192.168.10.49:19888</value>
>         </property>
>         <property>
>          <name>yarn.app.mapreduce.am.staging-dir</name>
>          <value>/data</value>
>         </property>
> .........
> <configuration>
> $ hadoop fs -ls /
> Found 6 items
> drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data
>
> $ hadoop fs -ls /data
> Found 3 items
> drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
> drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
> drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe
>
> [alex@hz23 ~]$ id
> uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)
>
>
> [alex@hz23 ~]$  hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2
> 100
> Number of Maps  = 2
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> 14/07/24 17:06:23 WARN security.UserGroupInformation:
> PriviledgedActionException as:alex (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
>         at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
>

RE: issue about run MR job use system user

Posted by java8964 <ja...@hotmail.com>.
Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be confirmed on the namenode?
Yong

Date: Thu, 24 Jul 2014 17:11:06 +0800
Subject: issue about run MR job use system user
From: justlooks@gmail.com
To: user@hadoop.apache.org

hi,maillist:
          i create a system user on a box of my hadoop cluster ,but when i run MR job use this user ,it get a problem, the /data directory is for mapreduce history server option, and i also add the user into hadoop group ,since the /data privilege is 775 ,so it can write by user in hadoop group,why still cause permssion error? anyone can help?

 

# useradd alex
 
<configuration>
        <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.address</name>
         <value>192.168.10.49:10020</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>192.168.10.49:19888</value>
        </property>

        <property>
         <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/data</value>
        </property>
.........
<configuration>

$ hadoop fs -ls /
Found 6 items
drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data

 
$ hadoop fs -ls /data
Found 3 items
drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe

 
[alex@hz23 ~]$ id
uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)

 
 
[alex@hz23 ~]$  hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2 100
Number of Maps  = 2
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Starting Job

14/07/24 17:06:23 WARN security.UserGroupInformation: PriviledgedActionException as:alex (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
 		 	   		  

RE: issue about run MR job use system user

Posted by java8964 <ja...@hotmail.com>.
Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be confirmed on the namenode?
Yong

Date: Thu, 24 Jul 2014 17:11:06 +0800
Subject: issue about run MR job use system user
From: justlooks@gmail.com
To: user@hadoop.apache.org

hi,maillist:
          i create a system user on a box of my hadoop cluster ,but when i run MR job use this user ,it get a problem, the /data directory is for mapreduce history server option, and i also add the user into hadoop group ,since the /data privilege is 775 ,so it can write by user in hadoop group,why still cause permssion error? anyone can help?

 

# useradd alex
 
<configuration>
        <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.address</name>
         <value>192.168.10.49:10020</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>192.168.10.49:19888</value>
        </property>

        <property>
         <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/data</value>
        </property>
.........
<configuration>

$ hadoop fs -ls /
Found 6 items
drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data

 
$ hadoop fs -ls /data
Found 3 items
drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe

 
[alex@hz23 ~]$ id
uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)

 
 
[alex@hz23 ~]$  hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2 100
Number of Maps  = 2
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Starting Job

14/07/24 17:06:23 WARN security.UserGroupInformation: PriviledgedActionException as:alex (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
 		 	   		  

RE: issue about run MR job use system user

Posted by java8964 <ja...@hotmail.com>.
Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be confirmed on the namenode?
Yong

Date: Thu, 24 Jul 2014 17:11:06 +0800
Subject: issue about run MR job use system user
From: justlooks@gmail.com
To: user@hadoop.apache.org

hi,maillist:
          i create a system user on a box of my hadoop cluster ,but when i run MR job use this user ,it get a problem, the /data directory is for mapreduce history server option, and i also add the user into hadoop group ,since the /data privilege is 775 ,so it can write by user in hadoop group,why still cause permssion error? anyone can help?

 

# useradd alex
 
<configuration>
        <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.address</name>
         <value>192.168.10.49:10020</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>192.168.10.49:19888</value>
        </property>

        <property>
         <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/data</value>
        </property>
.........
<configuration>

$ hadoop fs -ls /
Found 6 items
drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data

 
$ hadoop fs -ls /data
Found 3 items
drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe

 
[alex@hz23 ~]$ id
uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)

 
 
[alex@hz23 ~]$  hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2 100
Number of Maps  = 2
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Starting Job

14/07/24 17:06:23 WARN security.UserGroupInformation: PriviledgedActionException as:alex (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x
 		 	   		  

RE: issue about run MR job use system user

Posted by java8964 <ja...@hotmail.com>.
Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run command 'id alex' to prove it? And 'Alex' belongs to 'hadoop' group can be confirmed on the namenode?
Yong

Date: Thu, 24 Jul 2014 17:11:06 +0800
Subject: issue about run MR job use system user
From: justlooks@gmail.com
To: user@hadoop.apache.org

hi,maillist:
          i create a system user on a box of my hadoop cluster ,but when i run MR job use this user ,it get a problem, the /data directory is for mapreduce history server option, and i also add the user into hadoop group ,since the /data privilege is 775 ,so it can write by user in hadoop group,why still cause permssion error? anyone can help?

 

# useradd alex
 
<configuration>
        <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.address</name>
         <value>192.168.10.49:10020</value>
        </property>

        <property>
         <name>mapreduce.jobhistory.webapp.address</name>
         <value>192.168.10.49:19888</value>
        </property>

        <property>
         <name>yarn.app.mapreduce.am.staging-dir</name>
         <value>/data</value>
        </property>
.........
<configuration>

$ hadoop fs -ls /
Found 6 items
drwxrwxr-x   - hdfs  hadoop          0 2014-07-14 18:17 /data

 
$ hadoop fs -ls /data
Found 3 items
drwx------   - hdfs hadoop          0 2014-07-09 08:49 /data/hdfs
drwxrwxrwt   - hdfs hadoop          0 2014-07-08 18:52 /data/history
drwx------   - pipe hadoop          0 2014-07-14 18:17 /data/pipe

 
[alex@hz23 ~]$ id
uid=501(alex) gid=501(alex) groups=501(alex),497(hadoop)

 
 
[alex@hz23 ~]$  hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.0.2.jar pi 2 100
Number of Maps  = 2
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Starting Job

14/07/24 17:06:23 WARN security.UserGroupInformation: PriviledgedActionException as:alex (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5490)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5472)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5446)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3600)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3544)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:739)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

org.apache.hadoop.security.AccessControlException: Permission denied: user=alex, access=WRITE, inode="/data":hdfs:hadoop:drwxrwxr-x