You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marcos Sousa <fa...@marcossousa.com> on 2013/03/27 21:02:08 UTC
Hadoop Mapreduce fails with permission management enabled
I enabled the permission management in my hadoop cluster, but I'm facing a
problem sending jobs with pig. This is the scenario:
1 - I have hadoop/hadoop user
2 - I have myuserapp/myuserapp user that runs PIG script.
3 - We setup the path /myapp to be owned by myuserapp
4 - We set pig.temp.dir to /myapp/pig/tmp
But when we pig try to run the jobs we got the following error:
job_201303221059_0009 all_actions,filtered,raw_data DISTINCT
Message: Job failed! Error - Job initialization failed:
org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=realtime, access=EXECUTE,
inode="system":hadoop:supergroup:rwx------
Hadoop jobtracker requires this permission to statup it's server.
My hadoop policy looks like:
<property>
<name>security.client.datanode.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
</property>
<property>
<name>security.inter.tracker.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
</property>
<property>
<name>security.job.submission.protocol.acl</name>
<value>hadoop,myuserapp supergroup,myuserapp</value>
<property>
My hdfs-site.xml:
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>755</value>
</property>
<property>
<name>dfs.web.ugi</name>
<value>hadoop,supergroup</value>
</property>
My core site:
...
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
...
And finally my mapred-site.xml
...
<property>
<name>mapred.local.dir</name>
<value>/tmp/mapred</value>
</property>
<property>
<name>mapreduce.jobtracker.jobhistory.location</name>
<value>/opt/logs/hadoop/history</value>
</property>
<property>
<name>mapreduce.jobtracker.staging.root.dir</name>
<value>/user</value>
</property>
Is there a missing configuration? How can I deal with multiples users
running jobs in a restrict HDFS cluster?
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Marcos Sousa <fa...@marcossousa.com>.
I "solved" the problem using Capacity Scheduler, because I'm using 1.0.4
It is known issue solved in version 1.2.0 (
https://issues.apache.org/jira/browse/MAPREDUCE-4398).
On Thu, Mar 28, 2013 at 11:08 AM, Bertrand Dechoux <de...@gmail.com>wrote:
> Permission denied: user=*realtime*, access=EXECUTE, inode="*system*":*hadoop:**supergroup:rwx------*
>
>
> It seems like you tried to run a job with a user 'realtime' but this one
> has no access to the 'system' directory, which according to the right
> 'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
> else 'hadoop/supergroup' and this user does not like sharing '------'.
>
> I would guess that the 'system' directory is the last level of the
> mapred.system.dir.
>
> The setting should be changed according to your environment.
>
> Regards
>
> Bertrand
>
>
> On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
>
>> I enabled the permission management in my hadoop cluster, but I'm facing
>> a problem sending jobs with pig. This is the scenario:
>>
>> 1 - I have hadoop/hadoop user
>>
>> 2 - I have myuserapp/myuserapp user that runs PIG script.
>>
>> 3 - We setup the path /myapp to be owned by myuserapp
>>
>> 4 - We set pig.temp.dir to /myapp/pig/tmp
>>
>> But when we pig try to run the jobs we got the following error:
>>
>> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>>
>> Hadoop jobtracker requires this permission to statup it's server.
>>
>> My hadoop policy looks like:
>>
>> <property>
>> <name>security.client.datanode.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.inter.tracker.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.job.submission.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> <property>
>>
>> My hdfs-site.xml:
>>
>> <property>
>> <name>dfs.permissions</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.data.dir.perm</name>
>> <value>755</value>
>> </property>
>>
>> <property>
>> <name>dfs.web.ugi</name>
>> <value>hadoop,supergroup</value>
>> </property>
>>
>> My core site:
>>
>> ...
>> <property>
>> <name>hadoop.security.authorization</name>
>> <value>true</value>
>> </property>
>> ...
>>
>> And finally my mapred-site.xml
>>
>> ...
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/tmp/mapred</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobtracker.jobhistory.location</name>
>> <value>/opt/logs/hadoop/history</value>
>> </property>
>> <property>
>>
>>
>>
>> <name>mapreduce.jobtracker.staging.root.dir</name>
>>
>>
>>
>> <value>/user</value>
>>
>>
>>
>> </property>
>>
>> Is there a missing configuration? How can I deal with multiples users
>> running jobs in a restrict HDFS cluster?
>>
>>
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Marcos Sousa <fa...@marcossousa.com>.
I "solved" the problem using Capacity Scheduler, because I'm using 1.0.4
It is known issue solved in version 1.2.0 (
https://issues.apache.org/jira/browse/MAPREDUCE-4398).
On Thu, Mar 28, 2013 at 11:08 AM, Bertrand Dechoux <de...@gmail.com>wrote:
> Permission denied: user=*realtime*, access=EXECUTE, inode="*system*":*hadoop:**supergroup:rwx------*
>
>
> It seems like you tried to run a job with a user 'realtime' but this one
> has no access to the 'system' directory, which according to the right
> 'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
> else 'hadoop/supergroup' and this user does not like sharing '------'.
>
> I would guess that the 'system' directory is the last level of the
> mapred.system.dir.
>
> The setting should be changed according to your environment.
>
> Regards
>
> Bertrand
>
>
> On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
>
>> I enabled the permission management in my hadoop cluster, but I'm facing
>> a problem sending jobs with pig. This is the scenario:
>>
>> 1 - I have hadoop/hadoop user
>>
>> 2 - I have myuserapp/myuserapp user that runs PIG script.
>>
>> 3 - We setup the path /myapp to be owned by myuserapp
>>
>> 4 - We set pig.temp.dir to /myapp/pig/tmp
>>
>> But when we pig try to run the jobs we got the following error:
>>
>> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>>
>> Hadoop jobtracker requires this permission to statup it's server.
>>
>> My hadoop policy looks like:
>>
>> <property>
>> <name>security.client.datanode.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.inter.tracker.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.job.submission.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> <property>
>>
>> My hdfs-site.xml:
>>
>> <property>
>> <name>dfs.permissions</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.data.dir.perm</name>
>> <value>755</value>
>> </property>
>>
>> <property>
>> <name>dfs.web.ugi</name>
>> <value>hadoop,supergroup</value>
>> </property>
>>
>> My core site:
>>
>> ...
>> <property>
>> <name>hadoop.security.authorization</name>
>> <value>true</value>
>> </property>
>> ...
>>
>> And finally my mapred-site.xml
>>
>> ...
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/tmp/mapred</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobtracker.jobhistory.location</name>
>> <value>/opt/logs/hadoop/history</value>
>> </property>
>> <property>
>>
>>
>>
>> <name>mapreduce.jobtracker.staging.root.dir</name>
>>
>>
>>
>> <value>/user</value>
>>
>>
>>
>> </property>
>>
>> Is there a missing configuration? How can I deal with multiples users
>> running jobs in a restrict HDFS cluster?
>>
>>
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Marcos Sousa <fa...@marcossousa.com>.
I "solved" the problem using Capacity Scheduler, because I'm using 1.0.4
It is known issue solved in version 1.2.0 (
https://issues.apache.org/jira/browse/MAPREDUCE-4398).
On Thu, Mar 28, 2013 at 11:08 AM, Bertrand Dechoux <de...@gmail.com>wrote:
> Permission denied: user=*realtime*, access=EXECUTE, inode="*system*":*hadoop:**supergroup:rwx------*
>
>
> It seems like you tried to run a job with a user 'realtime' but this one
> has no access to the 'system' directory, which according to the right
> 'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
> else 'hadoop/supergroup' and this user does not like sharing '------'.
>
> I would guess that the 'system' directory is the last level of the
> mapred.system.dir.
>
> The setting should be changed according to your environment.
>
> Regards
>
> Bertrand
>
>
> On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
>
>> I enabled the permission management in my hadoop cluster, but I'm facing
>> a problem sending jobs with pig. This is the scenario:
>>
>> 1 - I have hadoop/hadoop user
>>
>> 2 - I have myuserapp/myuserapp user that runs PIG script.
>>
>> 3 - We setup the path /myapp to be owned by myuserapp
>>
>> 4 - We set pig.temp.dir to /myapp/pig/tmp
>>
>> But when we pig try to run the jobs we got the following error:
>>
>> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>>
>> Hadoop jobtracker requires this permission to statup it's server.
>>
>> My hadoop policy looks like:
>>
>> <property>
>> <name>security.client.datanode.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.inter.tracker.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.job.submission.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> <property>
>>
>> My hdfs-site.xml:
>>
>> <property>
>> <name>dfs.permissions</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.data.dir.perm</name>
>> <value>755</value>
>> </property>
>>
>> <property>
>> <name>dfs.web.ugi</name>
>> <value>hadoop,supergroup</value>
>> </property>
>>
>> My core site:
>>
>> ...
>> <property>
>> <name>hadoop.security.authorization</name>
>> <value>true</value>
>> </property>
>> ...
>>
>> And finally my mapred-site.xml
>>
>> ...
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/tmp/mapred</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobtracker.jobhistory.location</name>
>> <value>/opt/logs/hadoop/history</value>
>> </property>
>> <property>
>>
>>
>>
>> <name>mapreduce.jobtracker.staging.root.dir</name>
>>
>>
>>
>> <value>/user</value>
>>
>>
>>
>> </property>
>>
>> Is there a missing configuration? How can I deal with multiples users
>> running jobs in a restrict HDFS cluster?
>>
>>
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Marcos Sousa <fa...@marcossousa.com>.
I "solved" the problem using Capacity Scheduler, because I'm using 1.0.4
It is known issue solved in version 1.2.0 (
https://issues.apache.org/jira/browse/MAPREDUCE-4398).
On Thu, Mar 28, 2013 at 11:08 AM, Bertrand Dechoux <de...@gmail.com>wrote:
> Permission denied: user=*realtime*, access=EXECUTE, inode="*system*":*hadoop:**supergroup:rwx------*
>
>
> It seems like you tried to run a job with a user 'realtime' but this one
> has no access to the 'system' directory, which according to the right
> 'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
> else 'hadoop/supergroup' and this user does not like sharing '------'.
>
> I would guess that the 'system' directory is the last level of the
> mapred.system.dir.
>
> The setting should be changed according to your environment.
>
> Regards
>
> Bertrand
>
>
> On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
>
>> I enabled the permission management in my hadoop cluster, but I'm facing
>> a problem sending jobs with pig. This is the scenario:
>>
>> 1 - I have hadoop/hadoop user
>>
>> 2 - I have myuserapp/myuserapp user that runs PIG script.
>>
>> 3 - We setup the path /myapp to be owned by myuserapp
>>
>> 4 - We set pig.temp.dir to /myapp/pig/tmp
>>
>> But when we pig try to run the jobs we got the following error:
>>
>> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>>
>> Hadoop jobtracker requires this permission to statup it's server.
>>
>> My hadoop policy looks like:
>>
>> <property>
>> <name>security.client.datanode.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.inter.tracker.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> </property>
>> <property>
>> <name>security.job.submission.protocol.acl</name>
>> <value>hadoop,myuserapp supergroup,myuserapp</value>
>> <property>
>>
>> My hdfs-site.xml:
>>
>> <property>
>> <name>dfs.permissions</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.datanode.data.dir.perm</name>
>> <value>755</value>
>> </property>
>>
>> <property>
>> <name>dfs.web.ugi</name>
>> <value>hadoop,supergroup</value>
>> </property>
>>
>> My core site:
>>
>> ...
>> <property>
>> <name>hadoop.security.authorization</name>
>> <value>true</value>
>> </property>
>> ...
>>
>> And finally my mapred-site.xml
>>
>> ...
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/tmp/mapred</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.jobtracker.jobhistory.location</name>
>> <value>/opt/logs/hadoop/history</value>
>> </property>
>> <property>
>>
>>
>>
>> <name>mapreduce.jobtracker.staging.root.dir</name>
>>
>>
>>
>> <value>/user</value>
>>
>>
>>
>> </property>
>>
>> Is there a missing configuration? How can I deal with multiples users
>> running jobs in a restrict HDFS cluster?
>>
>>
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Bertrand Dechoux <de...@gmail.com>.
Permission denied: user=*realtime*, access=EXECUTE,
inode="*system*":*hadoop:**supergroup:rwx------*
It seems like you tried to run a job with a user 'realtime' but this one
has no access to the 'system' directory, which according to the right
'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
else 'hadoop/supergroup' and this user does not like sharing '------'.
I would guess that the 'system' directory is the last level of the
mapred.system.dir.
The setting should be changed according to your environment.
Regards
Bertrand
On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
> I enabled the permission management in my hadoop cluster, but I'm facing a
> problem sending jobs with pig. This is the scenario:
>
> 1 - I have hadoop/hadoop user
>
> 2 - I have myuserapp/myuserapp user that runs PIG script.
>
> 3 - We setup the path /myapp to be owned by myuserapp
>
> 4 - We set pig.temp.dir to /myapp/pig/tmp
>
> But when we pig try to run the jobs we got the following error:
>
> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>
> Hadoop jobtracker requires this permission to statup it's server.
>
> My hadoop policy looks like:
>
> <property>
> <name>security.client.datanode.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.inter.tracker.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.job.submission.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> <property>
>
> My hdfs-site.xml:
>
> <property>
> <name>dfs.permissions</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir.perm</name>
> <value>755</value>
> </property>
>
> <property>
> <name>dfs.web.ugi</name>
> <value>hadoop,supergroup</value>
> </property>
>
> My core site:
>
> ...
> <property>
> <name>hadoop.security.authorization</name>
> <value>true</value>
> </property>
> ...
>
> And finally my mapred-site.xml
>
> ...
> <property>
> <name>mapred.local.dir</name>
> <value>/tmp/mapred</value>
> </property>
>
> <property>
> <name>mapreduce.jobtracker.jobhistory.location</name>
> <value>/opt/logs/hadoop/history</value>
> </property>
> <property>
>
> <name>mapreduce.jobtracker.staging.root.dir</name>
>
> <value>/user</value>
>
> </property>
>
> Is there a missing configuration? How can I deal with multiples users
> running jobs in a restrict HDFS cluster?
>
>
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Bertrand Dechoux <de...@gmail.com>.
Permission denied: user=*realtime*, access=EXECUTE,
inode="*system*":*hadoop:**supergroup:rwx------*
It seems like you tried to run a job with a user 'realtime' but this one
has no access to the 'system' directory, which according to the right
'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
else 'hadoop/supergroup' and this user does not like sharing '------'.
I would guess that the 'system' directory is the last level of the
mapred.system.dir.
The setting should be changed according to your environment.
Regards
Bertrand
On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
> I enabled the permission management in my hadoop cluster, but I'm facing a
> problem sending jobs with pig. This is the scenario:
>
> 1 - I have hadoop/hadoop user
>
> 2 - I have myuserapp/myuserapp user that runs PIG script.
>
> 3 - We setup the path /myapp to be owned by myuserapp
>
> 4 - We set pig.temp.dir to /myapp/pig/tmp
>
> But when we pig try to run the jobs we got the following error:
>
> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>
> Hadoop jobtracker requires this permission to statup it's server.
>
> My hadoop policy looks like:
>
> <property>
> <name>security.client.datanode.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.inter.tracker.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.job.submission.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> <property>
>
> My hdfs-site.xml:
>
> <property>
> <name>dfs.permissions</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir.perm</name>
> <value>755</value>
> </property>
>
> <property>
> <name>dfs.web.ugi</name>
> <value>hadoop,supergroup</value>
> </property>
>
> My core site:
>
> ...
> <property>
> <name>hadoop.security.authorization</name>
> <value>true</value>
> </property>
> ...
>
> And finally my mapred-site.xml
>
> ...
> <property>
> <name>mapred.local.dir</name>
> <value>/tmp/mapred</value>
> </property>
>
> <property>
> <name>mapreduce.jobtracker.jobhistory.location</name>
> <value>/opt/logs/hadoop/history</value>
> </property>
> <property>
>
> <name>mapreduce.jobtracker.staging.root.dir</name>
>
> <value>/user</value>
>
> </property>
>
> Is there a missing configuration? How can I deal with multiples users
> running jobs in a restrict HDFS cluster?
>
>
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Bertrand Dechoux <de...@gmail.com>.
Permission denied: user=*realtime*, access=EXECUTE,
inode="*system*":*hadoop:**supergroup:rwx------*
It seems like you tried to run a job with a user 'realtime' but this one
has no access to the 'system' directory, which according to the right
'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
else 'hadoop/supergroup' and this user does not like sharing '------'.
I would guess that the 'system' directory is the last level of the
mapred.system.dir.
The setting should be changed according to your environment.
Regards
Bertrand
On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
> I enabled the permission management in my hadoop cluster, but I'm facing a
> problem sending jobs with pig. This is the scenario:
>
> 1 - I have hadoop/hadoop user
>
> 2 - I have myuserapp/myuserapp user that runs PIG script.
>
> 3 - We setup the path /myapp to be owned by myuserapp
>
> 4 - We set pig.temp.dir to /myapp/pig/tmp
>
> But when we pig try to run the jobs we got the following error:
>
> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>
> Hadoop jobtracker requires this permission to statup it's server.
>
> My hadoop policy looks like:
>
> <property>
> <name>security.client.datanode.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.inter.tracker.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.job.submission.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> <property>
>
> My hdfs-site.xml:
>
> <property>
> <name>dfs.permissions</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir.perm</name>
> <value>755</value>
> </property>
>
> <property>
> <name>dfs.web.ugi</name>
> <value>hadoop,supergroup</value>
> </property>
>
> My core site:
>
> ...
> <property>
> <name>hadoop.security.authorization</name>
> <value>true</value>
> </property>
> ...
>
> And finally my mapred-site.xml
>
> ...
> <property>
> <name>mapred.local.dir</name>
> <value>/tmp/mapred</value>
> </property>
>
> <property>
> <name>mapreduce.jobtracker.jobhistory.location</name>
> <value>/opt/logs/hadoop/history</value>
> </property>
> <property>
>
> <name>mapreduce.jobtracker.staging.root.dir</name>
>
> <value>/user</value>
>
> </property>
>
> Is there a missing configuration? How can I deal with multiples users
> running jobs in a restrict HDFS cluster?
>
>
Re: Hadoop Mapreduce fails with permission management enabled
Posted by Bertrand Dechoux <de...@gmail.com>.
Permission denied: user=*realtime*, access=EXECUTE,
inode="*system*":*hadoop:**supergroup:rwx------*
It seems like you tried to run a job with a user 'realtime' but this one
has no access to the 'system' directory, which according to the right
'hadoop:supergroup:rwx------' seems quite logical. It belongs to someone
else 'hadoop/supergroup' and this user does not like sharing '------'.
I would guess that the 'system' directory is the last level of the
mapred.system.dir.
The setting should be changed according to your environment.
Regards
Bertrand
On Wed, Mar 27, 2013 at 9:02 PM, Marcos Sousa <fa...@marcossousa.com>wrote:
> I enabled the permission management in my hadoop cluster, but I'm facing a
> problem sending jobs with pig. This is the scenario:
>
> 1 - I have hadoop/hadoop user
>
> 2 - I have myuserapp/myuserapp user that runs PIG script.
>
> 3 - We setup the path /myapp to be owned by myuserapp
>
> 4 - We set pig.temp.dir to /myapp/pig/tmp
>
> But when we pig try to run the jobs we got the following error:
>
> job_201303221059_0009 all_actions,filtered,raw_data DISTINCT Message: Job failed! Error - Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=realtime, access=EXECUTE, inode="system":hadoop:supergroup:rwx------
>
> Hadoop jobtracker requires this permission to statup it's server.
>
> My hadoop policy looks like:
>
> <property>
> <name>security.client.datanode.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.inter.tracker.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> </property>
> <property>
> <name>security.job.submission.protocol.acl</name>
> <value>hadoop,myuserapp supergroup,myuserapp</value>
> <property>
>
> My hdfs-site.xml:
>
> <property>
> <name>dfs.permissions</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.datanode.data.dir.perm</name>
> <value>755</value>
> </property>
>
> <property>
> <name>dfs.web.ugi</name>
> <value>hadoop,supergroup</value>
> </property>
>
> My core site:
>
> ...
> <property>
> <name>hadoop.security.authorization</name>
> <value>true</value>
> </property>
> ...
>
> And finally my mapred-site.xml
>
> ...
> <property>
> <name>mapred.local.dir</name>
> <value>/tmp/mapred</value>
> </property>
>
> <property>
> <name>mapreduce.jobtracker.jobhistory.location</name>
> <value>/opt/logs/hadoop/history</value>
> </property>
> <property>
>
> <name>mapreduce.jobtracker.staging.root.dir</name>
>
> <value>/user</value>
>
> </property>
>
> Is there a missing configuration? How can I deal with multiples users
> running jobs in a restrict HDFS cluster?
>
>