You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Clay B." <cw...@clayb.net> on 2012/07/16 21:05:55 UTC
Group mismatches?
Hi all,
I have a Hadoop cluster which uses Samba to map an Active Directory domain
to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with
groups. Does anyone have any debugging advice, or how to refresh the DFS
groups mapping? If not, should I file a bug at
https://issues.apache.org/jira/browse/HADOOP?
# I see the following error:
[clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
ls: could not get get listing for 'hdfs://hamster:9000/projects/foobarcommander' : org.apache.hadoop.security.AccessControlException: Permission denied: user=clayb, access=READ_EXECUTE, inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx---
# I verify group membership -- look a mismatch!
[clayb@hamster ~]$ which groups
/usr/bin/groups
[clayb@hamster ~]$ groups
foobarcommander xxx_rec_eng domain users all all_north america batchlogon xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users BUILTIN\users
[clayb@hamster ~]$ hadoop dfsgroups
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
Notice, in particular the foobarcommander group is only shown for my
/usr/bin/groups output. It looks like the following from the HDFS
Permissions Guide[1] is not correct, in my case:
"The group list is the equivalent of `bash -c groups`."
# I have tried the following to no useful effect:
[admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
# I do, however, see other users with the foobarcommander group, so the group should be "visible" to Hadoop:
[clayb@hamster ~]$ hadoop dfsgroups pat
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
pat : domain users all_north america all_san diego all foobarcommander BUILTIN\users
# And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me:
[clayb@hamster ~]$ hadoop mrgroups
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
# And I see that the system is getting the right data via getent(1):
[clayb@hamster ~]$ getent group foobarcommander
foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15
# I am using Cloudera's CDH3u4 Hadoop:
[clayb@hamster ~]$ hadoop version
Hadoop 0.20.2-cdh3u4
Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r 214dd731e3bdb687cb55988d3f47dd9e248c5690
Compiled by root on Mon May 7 14:03:02 PDT 2012
>From source with checksum a60c9795e41a3248b212344fb131c12c
I also do not see any obviously useful errors in my namenode logs.
-Clay
[1]:
http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity
Re: Group mismatches?
Posted by "Clay B." <cw...@clayb.net>.
Hi Edward,
I'm not sure I understand.
We all share the primary group domain_users (since these come from Active
Directory over Samba). Still, we are able to be in multiple projects by
using secondary groups, and it seems dfsgroups/mrgroups tries to report
secondary groups.
Regardless, if I use the newgrp(1) command to switch my primary group,
Hadoop still seems blind to the foobarcommander group, however:
[clayb@hamster ~]$ newgrp foobarcommander
[clayb@hamster ~]$ id
uid=16777217(clayb) gid=16777316(foobarcommander) groups=50000(xxx_rec_eng),16777216(domain users),16777218(all),16777221(all_north america),16777223(batchlogon),16777226(xxx-s),16777229(xxx03-s),16777235(xxx1-admins),16777236(xxx-emr-users),16777237(xxx-emr-admins),16777265(xxx1-users),16777285(BUILTIN\users),16777316(foobarcommander)
[clayb@hamster ~]$ hadoop dfsgroups
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
-Clay
On Mon, 16 Jul 2012, Edward Capriolo wrote:
> In all places I have found it only to be the primary group, not all
> the users supplemental groups.
>
> On Mon, Jul 16, 2012 at 3:05 PM, Clay B. <cw...@clayb.net> wrote:
>> Hi all,
>>
>> I have a Hadoop cluster which uses Samba to map an Active Directory domain
>> to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with
>> groups. Does anyone have any debugging advice, or how to refresh the DFS
>> groups mapping? If not, should I file a bug at
>> https://issues.apache.org/jira/browse/HADOOP?
>>
>> # I see the following error:
>> [clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander
>> log4j:ERROR Could not find value for key log4j.appender.NullAppender
>> log4j:ERROR Could not instantiate appender named "NullAppender".
>> ls: could not get get listing for
>> 'hdfs://hamster:9000/projects/foobarcommander' :
>> org.apache.hadoop.security.AccessControlException: Permission denied:
>> user=clayb, access=READ_EXECUTE,
>> inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx---
>>
>> # I verify group membership -- look a mismatch!
>> [clayb@hamster ~]$ which groups
>> /usr/bin/groups
>> [clayb@hamster ~]$ groups
>> foobarcommander xxx_rec_eng domain users all all_north america batchlogon
>> xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users
>> BUILTIN\users
>> [clayb@hamster ~]$ hadoop dfsgroups
>> log4j:ERROR Could not find value for key log4j.appender.NullAppender
>> log4j:ERROR Could not instantiate appender named "NullAppender".
>> clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon
>> all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
>>
>> Notice, in particular the foobarcommander group is only shown for my
>> /usr/bin/groups output. It looks like the following from the HDFS
>> Permissions Guide[1] is not correct, in my case:
>> "The group list is the equivalent of `bash -c groups`."
>>
>> # I have tried the following to no useful effect:
>> [admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings
>> log4j:ERROR Could not find value for key log4j.appender.NullAppender
>> log4j:ERROR Could not instantiate appender named "NullAppender".
>>
>> # I do, however, see other users with the foobarcommander group, so the
>> group should be "visible" to Hadoop:
>> [clayb@hamster ~]$ hadoop dfsgroups pat
>> log4j:ERROR Could not find value for key log4j.appender.NullAppender
>> log4j:ERROR Could not instantiate appender named "NullAppender".
>> pat : domain users all_north america all_san diego all foobarcommander
>> BUILTIN\users
>> # And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me:
>> [clayb@hamster ~]$ hadoop mrgroups
>> log4j:ERROR Could not find value for key log4j.appender.NullAppender
>> log4j:ERROR Could not instantiate appender named "NullAppender".
>> clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon
>> all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
>>
>> # And I see that the system is getting the right data via getent(1):
>> [clayb@hamster ~]$ getent group foobarcommander
>> foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15
>>
>> # I am using Cloudera's CDH3u4 Hadoop:
>> [clayb@hamster ~]$ hadoop version
>> Hadoop 0.20.2-cdh3u4
>> Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r
>> 214dd731e3bdb687cb55988d3f47dd9e248c5690
>> Compiled by root on Mon May 7 14:03:02 PDT 2012
>> From source with checksum a60c9795e41a3248b212344fb131c12c
>>
>> I also do not see any obviously useful errors in my namenode logs.
>>
>> -Clay
>>
>> [1]:
>> http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity
>>
>
Re: Group mismatches?
Posted by Edward Capriolo <ed...@gmail.com>.
In all places I have found it only to be the primary group, not all
the users supplemental groups.
On Mon, Jul 16, 2012 at 3:05 PM, Clay B. <cw...@clayb.net> wrote:
> Hi all,
>
> I have a Hadoop cluster which uses Samba to map an Active Directory domain
> to my CentOS 5.7 Hadoop cluster. However, I notice a strange mismatch with
> groups. Does anyone have any debugging advice, or how to refresh the DFS
> groups mapping? If not, should I file a bug at
> https://issues.apache.org/jira/browse/HADOOP?
>
> # I see the following error:
> [clayb@hamster ~]$ hadoop fs -ls /projects/foobarcommander
> log4j:ERROR Could not find value for key log4j.appender.NullAppender
> log4j:ERROR Could not instantiate appender named "NullAppender".
> ls: could not get get listing for
> 'hdfs://hamster:9000/projects/foobarcommander' :
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=clayb, access=READ_EXECUTE,
> inode="/projects/foobarcommander":hadmin:foobarcommander:drwxrwx---
>
> # I verify group membership -- look a mismatch!
> [clayb@hamster ~]$ which groups
> /usr/bin/groups
> [clayb@hamster ~]$ groups
> foobarcommander xxx_rec_eng domain users all all_north america batchlogon
> xxx-s xxx03-s xxx1-admins xxx-emr-users xxx-emr-admins xxx1-users
> BUILTIN\users
> [clayb@hamster ~]$ hadoop dfsgroups
> log4j:ERROR Could not find value for key log4j.appender.NullAppender
> log4j:ERROR Could not instantiate appender named "NullAppender".
> clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon
> all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
>
> Notice, in particular the foobarcommander group is only shown for my
> /usr/bin/groups output. It looks like the following from the HDFS
> Permissions Guide[1] is not correct, in my case:
> "The group list is the equivalent of `bash -c groups`."
>
> # I have tried the following to no useful effect:
> [admin@hamster ~]$ hadoop dfsadmin -refreshUserToGroupsMappings
> log4j:ERROR Could not find value for key log4j.appender.NullAppender
> log4j:ERROR Could not instantiate appender named "NullAppender".
>
> # I do, however, see other users with the foobarcommander group, so the
> group should be "visible" to Hadoop:
> [clayb@hamster ~]$ hadoop dfsgroups pat
> log4j:ERROR Could not find value for key log4j.appender.NullAppender
> log4j:ERROR Could not instantiate appender named "NullAppender".
> pat : domain users all_north america all_san diego all foobarcommander
> BUILTIN\users
> # And 'hadoop mrgroups' (like dfsgroups) returns the same bad data, for me:
> [clayb@hamster ~]$ hadoop mrgroups
> log4j:ERROR Could not find value for key log4j.appender.NullAppender
> log4j:ERROR Could not instantiate appender named "NullAppender".
> clayb : domain users xxx_rec_eng xxx-emr-users all xxx-emr-admins batchlogon
> all_north america xxx1-users xxx-s xxx03-s xxx1-admins BUILTIN\users
>
> # And I see that the system is getting the right data via getent(1):
> [clayb@hamster ~]$ getent group foobarcommander
> foobarcommander:*:16777316:pat,user1,user2,user3,clayb,user4,user5,user6,user7,user8,user9,user10,user12,user13,user14,user15
>
> # I am using Cloudera's CDH3u4 Hadoop:
> [clayb@hamster ~]$ hadoop version
> Hadoop 0.20.2-cdh3u4
> Subversion file:///data/1/tmp/topdir/BUILD/hadoop-0.20.2-cdh3u4 -r
> 214dd731e3bdb687cb55988d3f47dd9e248c5690
> Compiled by root on Mon May 7 14:03:02 PDT 2012
> From source with checksum a60c9795e41a3248b212344fb131c12c
>
> I also do not see any obviously useful errors in my namenode logs.
>
> -Clay
>
> [1]:
> http://hadoop.apache.org/common/docs/r0.20.2/hdfs_permissions_guide.html#User+Identity
>