You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/12/10 09:06:48 UTC
multiusers in hadoop through LDAP
Hi
In my cluster ,I want to have multiusers for different purpose.The usual
method is to add a user through the OS on Hadoop NameNode .
I notice the hadoop also support to LDAP, could I add user through LDAP
instead through OS? So that if a user is authenticated by the LDAP ,who
will also access the HDFS directory?
Regards
Re: multiusers in hadoop through LDAP
Posted by YouPeng Yang <yy...@gmail.com>.
Hi
Thanks a lot for your replies.
I will try the LDAP+hadoop.security.group.mapping.ldap.*. Right now I can
not catch this question.
Regards.
2013/12/11 Jay Vyas <ja...@gmail.com>
> So, not knowing much about LDAP, but being very interested in the
> multiuser problem on multiuser filesystems, i was excited to see this
> question.... Im researching the same thing at the moment, and it seems
> obviated by the fact that :
>
> - the FileSystem API itslef provides implementations for getting group and
> user names / permissions....
>
> And furthermore
>
> - the linux task controllers launch jobs as the user submitting the job,
> whereas the regular task controllers launch tasksunder the YARN daemon
> name, iirc.
>
> So.... where does LDAP begin and TaskController / FileSystem notions of
> ownership end.... ?
>
> I guess I'm also asking what are the entites which are "ownable" in hadoop
> app , and how we can leverage the GroupMappingServiceProviders to deploy
> more flexible hadoop environments.
>
> Any thoughts on this would be appreciated.
>
> On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
>> Pandya suggests.
>>
>> =====
>>
>> In advance, just to share our story related to LDAP +
>> hadoop.security.group.mapping.ldap.*, if you run into the same
>> limitation as we did:
>>
>> In many cases hadoop.security.group.mapping.ldap.* should solve your
>> problem. Unfortunately, they did now work for us. The problematic
>> setting relates to an additional filter to use when searching for LDAP
>> groups. We wanted to use posixGroups filter, but it is currently not
>> supported by Hadoop. Finally, we found a workaround using name service
>> switch configuration where we specified that the LDAP should the primary
>> source of information about groups of our users. This means that we solved
>> this problem on the operating system level, not on Hadoop level.
>>
>> You can read more about this issue here:
>>
>> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
>> and here
>> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
>> 18-26).
>>
>>
>> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>>
>>>
>>> have you looked at hadoop.security.group.mapping.ldap.* in
>>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>>
>>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yypvsxf19870706@gmail.com
>>> > wrote:
>>>
>>>> Hi
>>>>
>>>> In my cluster ,I want to have multiusers for different purpose.The
>>>> usual method is to add a user through the OS on Hadoop NameNode .
>>>> I notice the hadoop also support to LDAP, could I add user through
>>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>>> ,who will also access the HDFS directory?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: multiusers in hadoop through LDAP
Posted by YouPeng Yang <yy...@gmail.com>.
Hi
Thanks a lot for your replies.
I will try the LDAP+hadoop.security.group.mapping.ldap.*. Right now I can
not catch this question.
Regards.
2013/12/11 Jay Vyas <ja...@gmail.com>
> So, not knowing much about LDAP, but being very interested in the
> multiuser problem on multiuser filesystems, i was excited to see this
> question.... Im researching the same thing at the moment, and it seems
> obviated by the fact that :
>
> - the FileSystem API itslef provides implementations for getting group and
> user names / permissions....
>
> And furthermore
>
> - the linux task controllers launch jobs as the user submitting the job,
> whereas the regular task controllers launch tasksunder the YARN daemon
> name, iirc.
>
> So.... where does LDAP begin and TaskController / FileSystem notions of
> ownership end.... ?
>
> I guess I'm also asking what are the entites which are "ownable" in hadoop
> app , and how we can leverage the GroupMappingServiceProviders to deploy
> more flexible hadoop environments.
>
> Any thoughts on this would be appreciated.
>
> On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
>> Pandya suggests.
>>
>> =====
>>
>> In advance, just to share our story related to LDAP +
>> hadoop.security.group.mapping.ldap.*, if you run into the same
>> limitation as we did:
>>
>> In many cases hadoop.security.group.mapping.ldap.* should solve your
>> problem. Unfortunately, they did now work for us. The problematic
>> setting relates to an additional filter to use when searching for LDAP
>> groups. We wanted to use posixGroups filter, but it is currently not
>> supported by Hadoop. Finally, we found a workaround using name service
>> switch configuration where we specified that the LDAP should the primary
>> source of information about groups of our users. This means that we solved
>> this problem on the operating system level, not on Hadoop level.
>>
>> You can read more about this issue here:
>>
>> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
>> and here
>> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
>> 18-26).
>>
>>
>> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>>
>>>
>>> have you looked at hadoop.security.group.mapping.ldap.* in
>>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>>
>>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yypvsxf19870706@gmail.com
>>> > wrote:
>>>
>>>> Hi
>>>>
>>>> In my cluster ,I want to have multiusers for different purpose.The
>>>> usual method is to add a user through the OS on Hadoop NameNode .
>>>> I notice the hadoop also support to LDAP, could I add user through
>>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>>> ,who will also access the HDFS directory?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: multiusers in hadoop through LDAP
Posted by YouPeng Yang <yy...@gmail.com>.
Hi
Thanks a lot for your replies.
I will try the LDAP+hadoop.security.group.mapping.ldap.*. Right now I can
not catch this question.
Regards.
2013/12/11 Jay Vyas <ja...@gmail.com>
> So, not knowing much about LDAP, but being very interested in the
> multiuser problem on multiuser filesystems, i was excited to see this
> question.... Im researching the same thing at the moment, and it seems
> obviated by the fact that :
>
> - the FileSystem API itslef provides implementations for getting group and
> user names / permissions....
>
> And furthermore
>
> - the linux task controllers launch jobs as the user submitting the job,
> whereas the regular task controllers launch tasksunder the YARN daemon
> name, iirc.
>
> So.... where does LDAP begin and TaskController / FileSystem notions of
> ownership end.... ?
>
> I guess I'm also asking what are the entites which are "ownable" in hadoop
> app , and how we can leverage the GroupMappingServiceProviders to deploy
> more flexible hadoop environments.
>
> Any thoughts on this would be appreciated.
>
> On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
>> Pandya suggests.
>>
>> =====
>>
>> In advance, just to share our story related to LDAP +
>> hadoop.security.group.mapping.ldap.*, if you run into the same
>> limitation as we did:
>>
>> In many cases hadoop.security.group.mapping.ldap.* should solve your
>> problem. Unfortunately, they did now work for us. The problematic
>> setting relates to an additional filter to use when searching for LDAP
>> groups. We wanted to use posixGroups filter, but it is currently not
>> supported by Hadoop. Finally, we found a workaround using name service
>> switch configuration where we specified that the LDAP should the primary
>> source of information about groups of our users. This means that we solved
>> this problem on the operating system level, not on Hadoop level.
>>
>> You can read more about this issue here:
>>
>> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
>> and here
>> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
>> 18-26).
>>
>>
>> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>>
>>>
>>> have you looked at hadoop.security.group.mapping.ldap.* in
>>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>>
>>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yypvsxf19870706@gmail.com
>>> > wrote:
>>>
>>>> Hi
>>>>
>>>> In my cluster ,I want to have multiusers for different purpose.The
>>>> usual method is to add a user through the OS on Hadoop NameNode .
>>>> I notice the hadoop also support to LDAP, could I add user through
>>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>>> ,who will also access the HDFS directory?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: multiusers in hadoop through LDAP
Posted by YouPeng Yang <yy...@gmail.com>.
Hi
Thanks a lot for your replies.
I will try the LDAP+hadoop.security.group.mapping.ldap.*. Right now I can
not catch this question.
Regards.
2013/12/11 Jay Vyas <ja...@gmail.com>
> So, not knowing much about LDAP, but being very interested in the
> multiuser problem on multiuser filesystems, i was excited to see this
> question.... Im researching the same thing at the moment, and it seems
> obviated by the fact that :
>
> - the FileSystem API itslef provides implementations for getting group and
> user names / permissions....
>
> And furthermore
>
> - the linux task controllers launch jobs as the user submitting the job,
> whereas the regular task controllers launch tasksunder the YARN daemon
> name, iirc.
>
> So.... where does LDAP begin and TaskController / FileSystem notions of
> ownership end.... ?
>
> I guess I'm also asking what are the entites which are "ownable" in hadoop
> app , and how we can leverage the GroupMappingServiceProviders to deploy
> more flexible hadoop environments.
>
> Any thoughts on this would be appreciated.
>
> On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
>
>> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
>> Pandya suggests.
>>
>> =====
>>
>> In advance, just to share our story related to LDAP +
>> hadoop.security.group.mapping.ldap.*, if you run into the same
>> limitation as we did:
>>
>> In many cases hadoop.security.group.mapping.ldap.* should solve your
>> problem. Unfortunately, they did now work for us. The problematic
>> setting relates to an additional filter to use when searching for LDAP
>> groups. We wanted to use posixGroups filter, but it is currently not
>> supported by Hadoop. Finally, we found a workaround using name service
>> switch configuration where we specified that the LDAP should the primary
>> source of information about groups of our users. This means that we solved
>> this problem on the operating system level, not on Hadoop level.
>>
>> You can read more about this issue here:
>>
>> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
>> and here
>> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
>> 18-26).
>>
>>
>> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>>
>>>
>>> have you looked at hadoop.security.group.mapping.ldap.* in
>>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>>
>>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yypvsxf19870706@gmail.com
>>> > wrote:
>>>
>>>> Hi
>>>>
>>>> In my cluster ,I want to have multiusers for different purpose.The
>>>> usual method is to add a user through the OS on Hadoop NameNode .
>>>> I notice the hadoop also support to LDAP, could I add user through
>>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>>> ,who will also access the HDFS directory?
>>>>
>>>>
>>>> Regards
>>>>
>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: multiusers in hadoop through LDAP
Posted by Jay Vyas <ja...@gmail.com>.
So, not knowing much about LDAP, but being very interested in the multiuser
problem on multiuser filesystems, i was excited to see this question.... Im
researching the same thing at the moment, and it seems obviated by the fact
that :
- the FileSystem API itslef provides implementations for getting group and
user names / permissions....
And furthermore
- the linux task controllers launch jobs as the user submitting the job,
whereas the regular task controllers launch tasksunder the YARN daemon
name, iirc.
So.... where does LDAP begin and TaskController / FileSystem notions of
ownership end.... ?
I guess I'm also asking what are the entites which are "ownable" in hadoop
app , and how we can leverage the GroupMappingServiceProviders to deploy
more flexible hadoop environments.
Any thoughts on this would be appreciated.
On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
> Pandya suggests.
>
> =====
>
> In advance, just to share our story related to LDAP +
> hadoop.security.group.mapping.ldap.*, if you run into the same limitation
> as we did:
>
> In many cases hadoop.security.group.mapping.ldap.* should solve your
> problem. Unfortunately, they did now work for us. The problematic setting
> relates to an additional filter to use when searching for LDAP groups. We
> wanted to use posixGroups filter, but it is currently not supported by
> Hadoop. Finally, we found a workaround using name service switch
> configuration where we specified that the LDAP should the primary source of
> information about groups of our users. This means that we solved this
> problem on the operating system level, not on Hadoop level.
>
> You can read more about this issue here:
>
> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
> and here
> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
> 18-26).
>
>
> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>
>>
>> have you looked at hadoop.security.group.mapping.ldap.* in
>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>
>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>>
>>> Hi
>>>
>>> In my cluster ,I want to have multiusers for different purpose.The
>>> usual method is to add a user through the OS on Hadoop NameNode .
>>> I notice the hadoop also support to LDAP, could I add user through
>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>> ,who will also access the HDFS directory?
>>>
>>>
>>> Regards
>>>
>>
>>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: multiusers in hadoop through LDAP
Posted by Jay Vyas <ja...@gmail.com>.
So, not knowing much about LDAP, but being very interested in the multiuser
problem on multiuser filesystems, i was excited to see this question.... Im
researching the same thing at the moment, and it seems obviated by the fact
that :
- the FileSystem API itslef provides implementations for getting group and
user names / permissions....
And furthermore
- the linux task controllers launch jobs as the user submitting the job,
whereas the regular task controllers launch tasksunder the YARN daemon
name, iirc.
So.... where does LDAP begin and TaskController / FileSystem notions of
ownership end.... ?
I guess I'm also asking what are the entites which are "ownable" in hadoop
app , and how we can leverage the GroupMappingServiceProviders to deploy
more flexible hadoop environments.
Any thoughts on this would be appreciated.
On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
> Pandya suggests.
>
> =====
>
> In advance, just to share our story related to LDAP +
> hadoop.security.group.mapping.ldap.*, if you run into the same limitation
> as we did:
>
> In many cases hadoop.security.group.mapping.ldap.* should solve your
> problem. Unfortunately, they did now work for us. The problematic setting
> relates to an additional filter to use when searching for LDAP groups. We
> wanted to use posixGroups filter, but it is currently not supported by
> Hadoop. Finally, we found a workaround using name service switch
> configuration where we specified that the LDAP should the primary source of
> information about groups of our users. This means that we solved this
> problem on the operating system level, not on Hadoop level.
>
> You can read more about this issue here:
>
> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
> and here
> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
> 18-26).
>
>
> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>
>>
>> have you looked at hadoop.security.group.mapping.ldap.* in
>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>
>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>>
>>> Hi
>>>
>>> In my cluster ,I want to have multiusers for different purpose.The
>>> usual method is to add a user through the OS on Hadoop NameNode .
>>> I notice the hadoop also support to LDAP, could I add user through
>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>> ,who will also access the HDFS directory?
>>>
>>>
>>> Regards
>>>
>>
>>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: multiusers in hadoop through LDAP
Posted by Jay Vyas <ja...@gmail.com>.
So, not knowing much about LDAP, but being very interested in the multiuser
problem on multiuser filesystems, i was excited to see this question.... Im
researching the same thing at the moment, and it seems obviated by the fact
that :
- the FileSystem API itslef provides implementations for getting group and
user names / permissions....
And furthermore
- the linux task controllers launch jobs as the user submitting the job,
whereas the regular task controllers launch tasksunder the YARN daemon
name, iirc.
So.... where does LDAP begin and TaskController / FileSystem notions of
ownership end.... ?
I guess I'm also asking what are the entites which are "ownable" in hadoop
app , and how we can leverage the GroupMappingServiceProviders to deploy
more flexible hadoop environments.
Any thoughts on this would be appreciated.
On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
> Pandya suggests.
>
> =====
>
> In advance, just to share our story related to LDAP +
> hadoop.security.group.mapping.ldap.*, if you run into the same limitation
> as we did:
>
> In many cases hadoop.security.group.mapping.ldap.* should solve your
> problem. Unfortunately, they did now work for us. The problematic setting
> relates to an additional filter to use when searching for LDAP groups. We
> wanted to use posixGroups filter, but it is currently not supported by
> Hadoop. Finally, we found a workaround using name service switch
> configuration where we specified that the LDAP should the primary source of
> information about groups of our users. This means that we solved this
> problem on the operating system level, not on Hadoop level.
>
> You can read more about this issue here:
>
> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
> and here
> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
> 18-26).
>
>
> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>
>>
>> have you looked at hadoop.security.group.mapping.ldap.* in
>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>
>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>>
>>> Hi
>>>
>>> In my cluster ,I want to have multiusers for different purpose.The
>>> usual method is to add a user through the OS on Hadoop NameNode .
>>> I notice the hadoop also support to LDAP, could I add user through
>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>> ,who will also access the HDFS directory?
>>>
>>>
>>> Regards
>>>
>>
>>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: multiusers in hadoop through LDAP
Posted by Jay Vyas <ja...@gmail.com>.
So, not knowing much about LDAP, but being very interested in the multiuser
problem on multiuser filesystems, i was excited to see this question.... Im
researching the same thing at the moment, and it seems obviated by the fact
that :
- the FileSystem API itslef provides implementations for getting group and
user names / permissions....
And furthermore
- the linux task controllers launch jobs as the user submitting the job,
whereas the regular task controllers launch tasksunder the YARN daemon
name, iirc.
So.... where does LDAP begin and TaskController / FileSystem notions of
ownership end.... ?
I guess I'm also asking what are the entites which are "ownable" in hadoop
app , and how we can leverage the GroupMappingServiceProviders to deploy
more flexible hadoop environments.
Any thoughts on this would be appreciated.
On Tue, Dec 10, 2013 at 6:38 PM, Adam Kawa <ka...@gmail.com> wrote:
> Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
> Pandya suggests.
>
> =====
>
> In advance, just to share our story related to LDAP +
> hadoop.security.group.mapping.ldap.*, if you run into the same limitation
> as we did:
>
> In many cases hadoop.security.group.mapping.ldap.* should solve your
> problem. Unfortunately, they did now work for us. The problematic setting
> relates to an additional filter to use when searching for LDAP groups. We
> wanted to use posixGroups filter, but it is currently not supported by
> Hadoop. Finally, we found a workaround using name service switch
> configuration where we specified that the LDAP should the primary source of
> information about groups of our users. This means that we solved this
> problem on the operating system level, not on Hadoop level.
>
> You can read more about this issue here:
>
> http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
> and here
> http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013 (slides
> 18-26).
>
>
> 2013/12/10 Hardik Pandya <sm...@gmail.com>
>
>>
>> have you looked at hadoop.security.group.mapping.ldap.* in
>> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>>
>> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>>
>>> Hi
>>>
>>> In my cluster ,I want to have multiusers for different purpose.The
>>> usual method is to add a user through the OS on Hadoop NameNode .
>>> I notice the hadoop also support to LDAP, could I add user through
>>> LDAP instead through OS? So that if a user is authenticated by the LDAP
>>> ,who will also access the HDFS directory?
>>>
>>>
>>> Regards
>>>
>>
>>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: multiusers in hadoop through LDAP
Posted by Adam Kawa <ka...@gmail.com>.
Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
Pandya suggests.
=====
In advance, just to share our story related to LDAP +
hadoop.security.group.mapping.ldap.*, if you run into the same limitation
as we did:
In many cases hadoop.security.group.mapping.ldap.* should solve your
problem. Unfortunately, they did now work for us. The problematic setting
relates to an additional filter to use when searching for LDAP groups. We
wanted to use posixGroups filter, but it is currently not supported by
Hadoop. Finally, we found a workaround using name service switch
configuration where we specified that the LDAP should the primary source of
information about groups of our users. This means that we solved this
problem on the operating system level, not on Hadoop level.
You can read more about this issue here:
http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
and here
http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013
(slides
18-26).
2013/12/10 Hardik Pandya <sm...@gmail.com>
>
> have you looked at hadoop.security.group.mapping.ldap.* in
> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>
> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>
>
>
>
>
>
> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi
>>
>> In my cluster ,I want to have multiusers for different purpose.The
>> usual method is to add a user through the OS on Hadoop NameNode .
>> I notice the hadoop also support to LDAP, could I add user through LDAP
>> instead through OS? So that if a user is authenticated by the LDAP ,who
>> will also access the HDFS directory?
>>
>>
>> Regards
>>
>
>
Re: multiusers in hadoop through LDAP
Posted by Adam Kawa <ka...@gmail.com>.
Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
Pandya suggests.
=====
In advance, just to share our story related to LDAP +
hadoop.security.group.mapping.ldap.*, if you run into the same limitation
as we did:
In many cases hadoop.security.group.mapping.ldap.* should solve your
problem. Unfortunately, they did now work for us. The problematic setting
relates to an additional filter to use when searching for LDAP groups. We
wanted to use posixGroups filter, but it is currently not supported by
Hadoop. Finally, we found a workaround using name service switch
configuration where we specified that the LDAP should the primary source of
information about groups of our users. This means that we solved this
problem on the operating system level, not on Hadoop level.
You can read more about this issue here:
http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
and here
http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013
(slides
18-26).
2013/12/10 Hardik Pandya <sm...@gmail.com>
>
> have you looked at hadoop.security.group.mapping.ldap.* in
> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>
> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>
>
>
>
>
>
> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi
>>
>> In my cluster ,I want to have multiusers for different purpose.The
>> usual method is to add a user through the OS on Hadoop NameNode .
>> I notice the hadoop also support to LDAP, could I add user through LDAP
>> instead through OS? So that if a user is authenticated by the LDAP ,who
>> will also access the HDFS directory?
>>
>>
>> Regards
>>
>
>
Re: multiusers in hadoop through LDAP
Posted by Adam Kawa <ka...@gmail.com>.
Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
Pandya suggests.
=====
In advance, just to share our story related to LDAP +
hadoop.security.group.mapping.ldap.*, if you run into the same limitation
as we did:
In many cases hadoop.security.group.mapping.ldap.* should solve your
problem. Unfortunately, they did now work for us. The problematic setting
relates to an additional filter to use when searching for LDAP groups. We
wanted to use posixGroups filter, but it is currently not supported by
Hadoop. Finally, we found a workaround using name service switch
configuration where we specified that the LDAP should the primary source of
information about groups of our users. This means that we solved this
problem on the operating system level, not on Hadoop level.
You can read more about this issue here:
http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
and here
http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013
(slides
18-26).
2013/12/10 Hardik Pandya <sm...@gmail.com>
>
> have you looked at hadoop.security.group.mapping.ldap.* in
> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>
> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>
>
>
>
>
>
> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi
>>
>> In my cluster ,I want to have multiusers for different purpose.The
>> usual method is to add a user through the OS on Hadoop NameNode .
>> I notice the hadoop also support to LDAP, could I add user through LDAP
>> instead through OS? So that if a user is authenticated by the LDAP ,who
>> will also access the HDFS directory?
>>
>>
>> Regards
>>
>
>
Re: multiusers in hadoop through LDAP
Posted by Adam Kawa <ka...@gmail.com>.
Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
Pandya suggests.
=====
In advance, just to share our story related to LDAP +
hadoop.security.group.mapping.ldap.*, if you run into the same limitation
as we did:
In many cases hadoop.security.group.mapping.ldap.* should solve your
problem. Unfortunately, they did now work for us. The problematic setting
relates to an additional filter to use when searching for LDAP groups. We
wanted to use posixGroups filter, but it is currently not supported by
Hadoop. Finally, we found a workaround using name service switch
configuration where we specified that the LDAP should the primary source of
information about groups of our users. This means that we solved this
problem on the operating system level, not on Hadoop level.
You can read more about this issue here:
http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/
and here
http://www.slideshare.net/AdamKawa/hadoop-adventures-at-spotify-strata-conference-hadoop-world-2013
(slides
18-26).
2013/12/10 Hardik Pandya <sm...@gmail.com>
>
> have you looked at hadoop.security.group.mapping.ldap.* in
> hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
>
> additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may help
>
>
>
>
>
>
> On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi
>>
>> In my cluster ,I want to have multiusers for different purpose.The
>> usual method is to add a user through the OS on Hadoop NameNode .
>> I notice the hadoop also support to LDAP, could I add user through LDAP
>> instead through OS? So that if a user is authenticated by the LDAP ,who
>> will also access the HDFS directory?
>>
>>
>> Regards
>>
>
>
Re: multiusers in hadoop through LDAP
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked at hadoop.security.group.mapping.ldap.* in
hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may
help
On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi
>
> In my cluster ,I want to have multiusers for different purpose.The usual
> method is to add a user through the OS on Hadoop NameNode .
> I notice the hadoop also support to LDAP, could I add user through LDAP
> instead through OS? So that if a user is authenticated by the LDAP ,who
> will also access the HDFS directory?
>
>
> Regards
>
Re: multiusers in hadoop through LDAP
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked at hadoop.security.group.mapping.ldap.* in
hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may
help
On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi
>
> In my cluster ,I want to have multiusers for different purpose.The usual
> method is to add a user through the OS on Hadoop NameNode .
> I notice the hadoop also support to LDAP, could I add user through LDAP
> instead through OS? So that if a user is authenticated by the LDAP ,who
> will also access the HDFS directory?
>
>
> Regards
>
Re: multiusers in hadoop through LDAP
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked at hadoop.security.group.mapping.ldap.* in
hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may
help
On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi
>
> In my cluster ,I want to have multiusers for different purpose.The usual
> method is to add a user through the OS on Hadoop NameNode .
> I notice the hadoop also support to LDAP, could I add user through LDAP
> instead through OS? So that if a user is authenticated by the LDAP ,who
> will also access the HDFS directory?
>
>
> Regards
>
Re: multiusers in hadoop through LDAP
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked at hadoop.security.group.mapping.ldap.* in
hadoop-common/core-default.xml<http://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml>
additional resource<http://hakunamapdata.com/a-user-having-surprising-troubles-running-more-resource-intensive-hive-queries/>may
help
On Tue, Dec 10, 2013 at 3:06 AM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi
>
> In my cluster ,I want to have multiusers for different purpose.The usual
> method is to add a user through the OS on Hadoop NameNode .
> I notice the hadoop also support to LDAP, could I add user through LDAP
> instead through OS? So that if a user is authenticated by the LDAP ,who
> will also access the HDFS directory?
>
>
> Regards
>