You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Hafiz Mujadid <ha...@gmail.com> on 2015/12/01 22:31:30 UTC
Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop
permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Bosco, I have tried both mysql db and solr as well, only plugin related
> auditing is being shown
>
> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
> wrote:
>
>> Yes, you should fix audit first. That will help in debugging these issues
>> also.
>>
>> BTW, are you using Solr or DB?
>>
>> Recommendation is to use Solr. Yesterday, I have uploaded a new package
>> for setting up Solr. It is available as attachment in
>> https://issues.apache.org/jira/browse/RANGER-728. The instructions are
>> in
>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>
>> Give it a try.
>>
>> Thanks
>>
>> Bosco
>>
>>
>> From: Madhan Neethiraj <mn...@hortonworks.com>
>> Reply-To: <us...@ranger.incubator.apache.org>
>> Date: Monday, November 30, 2015 at 8:57 AM
>>
>> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Hafiz,
>>
>> Few things to check:
>> 1. Do you have another policy in Ranger that allows WRITE access?
>> 2. Can you disable this policy and try mkdir?
>>
>> Fixing the issue with audit will help; audit log will have the details of
>> how the access was allowed (hadoop-acl or ranger-acl; in case of
>> ranger-acl, the policy-ID that determined the access).
>>
>> Madhan
>>
>> From: Hafiz Mujadid <ha...@gmail.com>
>> Reply-To: "user@ranger.incubator.apache.org" <
>> user@ranger.incubator.apache.org>
>> Date: Monday, November 30, 2015 at 6:16 AM
>> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Bosco,
>>
>> I have followed above steps
>>
>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>> 2. changed the umask so newly created folder or files have following
>> permissions
>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>> 3. i changed the ownership of all folders in hdfs with hduser:hadoop
>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>
>>
>> but still group level permissions are not working.
>>
>> my audits are not working, i am trying to figure out the issue with
>> audits. i will let you know when audits are available.
>>
>>
>> thanks
>>
>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com>
>> wrote:
>>
>>> Bosco,
>>>
>>> I have followed above steps
>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>> changed the umask so newly created folder or files have following
>>> permissions
>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>
>>> but still group level permissions are not working.
>>>
>>>
>>> my audits are not working, i am trying to figure out the issue with
>>> audits. i will let you know when audits are available.
>>>
>>>
>>> thanks
>>>
>>>
>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> Can you check Ranger Audits?
>>>>
>>>> Also, do couple of things:
>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>
>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>
>>>> Bosco
>>>>
>>>>
>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>> To: <us...@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Yes Bosco, directory is being created.
>>>>
>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> What is happening here? Is the directory getting created?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Group level permission are not working in ranger
>>>>>
>>>>> Hi all
>>>>>
>>>>> I am trying to apply permission on an ldap group but it's not working
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>> But when i run following command
>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>
>>>>> i works successfully
>>>>> what is the issue? ldap users and groups are synced correctly as when
>>>>> i run the command *hdfs groups asma* it returns correct group
>>>>> asma : datascientist
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Thanks Madhan :)
On Fri, Dec 4, 2015 at 1:43 AM, Madhan Neethiraj <ma...@apache.org> wrote:
> Hafiz,
>
> The policy allows only READ access for user ‘mike’; since WRITE and
> EXECUTE are excluded (from allow), the policy does not make any decision
> for these access-types. Hence Ranger falls back to hadoop-acl to determine
> the authorization.
>
> If you would like to explicitly deny WRITE and EXECUTE access for user
> ‘mike’, please add a deny condition for this. Then you will see that WRITE
> and EXECUTE accesses were denied by Ranger.
>
> Thanks,
> Madhan
>
>
> From: Hafiz Mujadid <ha...@gmail.com>
> Reply-To: "user@ranger.incubator.apache.org" <
> user@ranger.incubator.apache.org>
> Date: Thursday, December 3, 2015 at 12:26 PM
>
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Re: Group level permission are not working in ranger
>
> Yes Bosco,
> I will write then to wiki on this weekend
>
> Thanks
>
> On Fri, Dec 4, 2015 at 1:25 AM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Hi Bosco!
>> When i run following command
>> HADOOP_USER_NAME=mike hdfs dfs -ls /perm
>>
>> it is showing allowed because of hadoop_ACL, Why it is not because of
>> ranger-acl?
>>
>> - 12/04/2015 01:22:28 AM mike
>> hadoopdev
>> hdfs
>> /perm READ_EXECUTEAllowed hadoop-acl 192.168.23.126 1
>>
>> On Fri, Dec 4, 2015 at 1:21 AM, Hafiz Mujadid <ha...@gmail.com>
>> wrote:
>>
>>> Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
>>>
>>>
>>> 12/04/2015 01:19:16 AM mike
>>> hadoopdev
>>> hdfs
>>> /perm/m2 WRITEDenied hadoop-acl 192.168.23.126 1
>>>
>>> On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com>
>>> wrote:
>>>
>>>> Yes, Bosco, it is denied , i made the mistake when i ran this test case
>>>> i have changed mike permission to read and execute. Test_Id 7 is Denied
>>>>
>>>> Thanks
>>>>
>>>> On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> Can we check the audit log? If Ranger is giving the write permission,
>>>>> then it is a bug.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>> From: Aneela Saleem <an...@platalytics.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Thursday, December 3, 2015 at 12:08 PM
>>>>>
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Agreed with Bosco
>>>>>
>>>>> On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hafiz, thanks for testing it.
>>>>>>
>>>>>> Regard test case #7, shouldn't Mike been denied?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Thursday, December 3, 2015 at 11:54 AM
>>>>>>
>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Hi Bosco*,*
>>>>>>
>>>>>> I have tested Deny, Allow , Exclude from deny and exclude from Allow
>>>>>> permissions. Following are details of my findings.
>>>>>>
>>>>>> *Test Cases*
>>>>>>
>>>>>> *Developer Group: *Roger, Smith
>>>>>>
>>>>>> *Data Scientist Group: *Mike, Clark
>>>>>>
>>>>>> Hadoop Resource : */perm*
>>>>>> Ranger Policy
>>>>>> [image: Inline image 2]
>>>>>>
>>>>>> Developers can do nothing having no read, write and execute
>>>>>> permissions except Roger who has full permissions .All users of group
>>>>>> Data-Scientist have permissions read, write and execute permissions
>>>>>> except *mike* who can't write.
>>>>>> Test Cases
>>>>>>
>>>>>> Test_ID
>>>>>>
>>>>>> User
>>>>>>
>>>>>> Group
>>>>>>
>>>>>> Command
>>>>>>
>>>>>> Expected
>>>>>>
>>>>>> Actual
>>>>>>
>>>>>> 1
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>> Developers
>>>>>>
>>>>>> Hdfs dfs -ls /perm
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> 2
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>> Developers
>>>>>>
>>>>>> Hdfs dfs -mkdir /perm/r
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> 3
>>>>>>
>>>>>> Smith
>>>>>>
>>>>>> Developers
>>>>>>
>>>>>> Hdfs dfs -mkdir /perm/s
>>>>>>
>>>>>> Denied
>>>>>>
>>>>>> Denied
>>>>>>
>>>>>> 4
>>>>>>
>>>>>> Smith
>>>>>>
>>>>>> Developers
>>>>>>
>>>>>> Hdfs dfs -ls /perm
>>>>>>
>>>>>> Denied
>>>>>>
>>>>>> Denied
>>>>>>
>>>>>> 5
>>>>>>
>>>>>> Clark
>>>>>>
>>>>>> Data-Scientis
>>>>>>
>>>>>> Hdfs dfs -ls /perm
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> 6
>>>>>>
>>>>>> Clark
>>>>>>
>>>>>> Data-Scientis
>>>>>>
>>>>>> Hdfs dfs -mkdir /perm/c
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> 7
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Data-Scientis
>>>>>>
>>>>>> Hdfs dfs -mkdir /perm/m
>>>>>>
>>>>>> Alowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>> 8
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Data-Scientis
>>>>>>
>>>>>> Hdfs dfs -ls /perm
>>>>>> Allowed
>>>>>>
>>>>>> Allowed
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <
>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Bosco,
>>>>>>>
>>>>>>> Thanks for your response, I am testing new feature of ranger
>>>>>>> Deny,Allow. will send you my findings in short.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> >I want to know why audits are showing that it is because of
>>>>>>>> hadoop-acl not ranger-acl?
>>>>>>>> Hafiz, this is a good question and we should probably document it
>>>>>>>> or come with a blog for this.
>>>>>>>>
>>>>>>>> Only for HDFS and YARN, we support falling back to native
>>>>>>>> permission check if we don’t have corresponding permission in Ranger. So in
>>>>>>>> your case, since there were no permissions in Ranger for “asma” to the
>>>>>>>> folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t
>>>>>>>> have native posix ACL for asma for the folder /mjd, it denied it. Since
>>>>>>>> hadoop was the last one to deny, you saw “hadoop-acl” in the audit record.
>>>>>>>> If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have
>>>>>>>> allowed creating the folder and the audit would should that hadoop-acl
>>>>>>>> allowed to create the folder.
>>>>>>>>
>>>>>>>> This also answers yours previous question why we want to make
>>>>>>>> umask=077 and chmod –r 000 to all application folders to be managed by
>>>>>>>> Ranger. So if there are no Ranger policies, then we want to hadoop also to
>>>>>>>> deny.
>>>>>>>>
>>>>>>>> With the recent deny feature, you can explicitly “deny” “asma” or
>>>>>>>> any group from creating/writing. Or you could deny all, but exclude
>>>>>>>> “developer’ and “sadaf” from the deny users.
>>>>>>>>
>>>>>>>> In the future release, I feel, we should provide a way to mark
>>>>>>>> certain folders to be managed exclusively by Ranger. And that will remove a
>>>>>>>> lot of confusion and also make the policy management more predictable.
>>>>>>>>
>>>>>>>> Does it answer your question?
>>>>>>>>
>>>>>>>> Bosco
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>>>>>>
>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Hi Bosco!
>>>>>>>>
>>>>>>>> I created a directory /mjd with following permissions
>>>>>>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>>>>>>
>>>>>>>> Then i made a policy with following permissions
>>>>>>>> [image: Inline image 1]
>>>>>>>> Datascientist group has one user asma and developer group has one
>>>>>>>> user named haniya and sadaf has no group.
>>>>>>>>
>>>>>>>> So when i run following command
>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>>>>>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>>>>>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *And audit of this command is as follow* Service Policy IDEvent
>>>>>>>> TimeUser Name / Type Resource Name Access TypeResult Access
>>>>>>>> Enforcer Client IP Event Count -- 12/02/2015 09:46:23 AM asma
>>>>>>>> hdfsRepo
>>>>>>>> /mjd/a1 WRITEDenied hadoop-acl 192.168.23.105 1
>>>>>>>> I want to know why audits are showing that it is because of
>>>>>>>> hadoop-acl not ranger-acl?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> You don’t need to. Since auditing is working, you can check who
>>>>>>>>> gave the permission without 000
>>>>>>>>>
>>>>>>>>> We recommend giving 000 at HDFS level, because Ranger by default
>>>>>>>>> falls back to HDFS permission. So for all folders you want to Ranger to be
>>>>>>>>> exclusive, you give as minimal permission as possible.
>>>>>>>>>
>>>>>>>>> I think, we should also make it configurable in Ranger. Where you
>>>>>>>>> can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you
>>>>>>>>> don’t have to worry about HDFS level ACLs.
>>>>>>>>>
>>>>>>>>> The reason you don’t want Ranger to manage everything because
>>>>>>>>> there are folders like tmp and user folders which want the system and user
>>>>>>>>> to manage themselves. But for application folders like Hive warehouse, you
>>>>>>>>> should let Ranger manage it.
>>>>>>>>>
>>>>>>>>> Bosco
>>>>>>>>>
>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>>>>>>
>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Bosco, I noticed group level permission works when we set hadoop
>>>>>>>>> permissions to 000. I am just curious why it is so ?
>>>>>>>>>
>>>>>>>>> is it always necessary to set hadoop permissions to 000 for ranger
>>>>>>>>> to work?
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>>
>>>>>>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>>>>>>> related auditing is being shown
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <
>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, you should fix audit first. That will help in debugging
>>>>>>>>>>> these issues also.
>>>>>>>>>>>
>>>>>>>>>>> BTW, are you using Solr or DB?
>>>>>>>>>>>
>>>>>>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>>>>>>> package for setting up Solr. It is available as attachment in
>>>>>>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The
>>>>>>>>>>> instructions are in
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>>>>>>
>>>>>>>>>>> Give it a try.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Bosco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>>>>>>
>>>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>>
>>>>>>>>>>> Hafiz,
>>>>>>>>>>>
>>>>>>>>>>> Few things to check:
>>>>>>>>>>> 1. Do you have another policy in Ranger that allows WRITE
>>>>>>>>>>> access?
>>>>>>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>>>>>>
>>>>>>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>>>>>>
>>>>>>>>>>> Madhan
>>>>>>>>>>>
>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>>
>>>>>>>>>>> Bosco,
>>>>>>>>>>>
>>>>>>>>>>> I have followed above steps
>>>>>>>>>>>
>>>>>>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49
>>>>>>>>>>> /pg
>>>>>>>>>>> 2. changed the umask so newly created folder or files have
>>>>>>>>>>> following permissions
>>>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>>>>>>> hduser:hadoop
>>>>>>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>>>
>>>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Bosco,
>>>>>>>>>>>>
>>>>>>>>>>>> I have followed above steps
>>>>>>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>>>>>> changed the umask so newly created folder or files have
>>>>>>>>>>>> following permissions
>>>>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>>>>> i changed the ownership of all folders in hdfs with
>>>>>>>>>>>> hduser:hadoop
>>>>>>>>>>>>
>>>>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <
>>>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Can you check Ranger Audits?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, do couple of things:
>>>>>>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>>>>
>>>>>>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bosco
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <
>>>>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Bosco
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>>>>>>> working
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But when i run following command
>>>>>>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i works successfully
>>>>>>>>>>>>>> what is the issue? ldap users and groups are synced correctly
>>>>>>>>>>>>>> as when i run the command *hdfs groups asma* it returns
>>>>>>>>>>>>>> correct group
>>>>>>>>>>>>>> asma : datascientist
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Madhan Neethiraj <ma...@apache.org>.
Hafiz,
The policy allows only READ access for user ‘mike’; since WRITE and EXECUTE are excluded (from allow), the policy does not make any decision for these access-types. Hence Ranger falls back to hadoop-acl to determine the authorization.
If you would like to explicitly deny WRITE and EXECUTE access for user ‘mike’, please add a deny condition for this. Then you will see that WRITE and EXECUTE accesses were denied by Ranger.
Thanks,
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 12:26 PM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco,
I will write then to wiki on this weekend
Thanks
On Fri, Dec 4, 2015 at 1:25 AM, Hafiz Mujadid <ha...@gmail.com> wrote:
Hi Bosco!
When i run following command
HADOOP_USER_NAME=mike hdfs dfs -ls /perm
it is showing allowed because of hadoop_ACL, Why it is not because of ranger-acl?
- 12/04/2015 01:22:28 AM mikehadoopdev
hdfs
/perm READ_EXECUTEAllowed hadoop-acl 192.168.23.126 1
On Fri, Dec 4, 2015 at 1:21 AM, Hafiz Mujadid <ha...@gmail.com> wrote:
Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
12/04/2015 01:19:16 AM mikehadoopdev
hdfs
/perm/m2 WRITEDenied hadoop-acl 192.168.23.126 1
On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com> wrote:
Yes, Bosco, it is denied , i made the mistake when i ran this test case i have changed mike permission to read and execute. Test_Id 7 is Denied
Thanks
On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can we check the audit log? If Ranger is giving the write permission, then it is a bug.
Thanks
Bosco
From: Aneela Saleem <an...@platalytics.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 12:08 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Agreed with Bosco
On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
Hafiz, thanks for testing it.
Regard test case #7, shouldn't Mike been denied?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 11:54 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco,
I have tested Deny, Allow , Exclude from deny and exclude from Allow permissions. Following are details of my findings.
Test Cases
Developer Group: Roger, Smith
Data Scientist Group: Mike, Clark
Hadoop Resource : /perm
Ranger Policy
Developers can do nothing having no read, write and execute permissions except Roger who has full permissions .All users of group Data-Scientist have permissions read, write and execute permissions except mike who can't write.
Test Cases
Test_IDUserGroupCommandExpectedActual
1RogerDevelopersHdfs dfs -ls /permAllowedAllowed
2RogerDevelopersHdfs dfs -mkdir /perm/rAllowedAllowed
3SmithDevelopersHdfs dfs -mkdir /perm/sDeniedDenied
4SmithDevelopersHdfs dfs -ls /permDeniedDenied
5ClarkData-ScientisHdfs dfs -ls /permAllowedAllowed
6ClarkData-ScientisHdfs dfs -mkdir /perm/cAllowedAllowed
7MikeData-ScientisHdfs dfs -mkdir /perm/mAlowedAllowed
8MikeData-ScientisHdfs dfs -ls /perm Allowed Allowed
On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Hi Bosco,
Thanks for your response, I am testing new feature of ranger Deny,Allow. will send you my findings in short.
Thanks
On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
>I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Hafiz, this is a good question and we should probably document it or come with a blog for this.
Only for HDFS and YARN, we support falling back to native permission check if we don’t have corresponding permission in Ranger. So in your case, since there were no permissions in Ranger for “asma” to the folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t have native posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed creating the folder and the audit would should that hadoop-acl allowed to create the folder.
This also answers yours previous question why we want to make umask=077 and chmod –r 000 to all application folders to be managed by Ranger. So if there are no Ranger policies, then we want to hadoop also to deny.
With the recent deny feature, you can explicitly “deny” “asma” or any group from creating/writing. Or you could deny all, but exclude “developer’ and “sadaf” from the deny users.
In the future release, I feel, we should provide a way to mark certain folders to be managed exclusively by Ranger. And that will remove a lot of confusion and also make the policy management more predictable.
Does it answer your question?
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 8:59 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco!
I created a directory /mjd with following permissions
drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd
Then i made a policy with following permissions
Datascientist group has one user asma and developer group has one user named haniya and sadaf has no group.
So when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1
mkdir: Permission denied: user=asma, access=WRITE, inode="/mjd/a1":hduser:supergroup:drwxr-xr-x
And audit of this command is as follow
Service
Policy IDEvent TimeUser Name / Type Resource Name Access TypeResult Access Enforcer Client IP Event Count
-- 12/02/2015 09:46:23 AM asmahdfsRepo
/mjd/a1 WRITEDenied hadoop-acl 192.168.23.105 1
I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Yes Bosco,
I will write then to wiki on this weekend
Thanks
On Fri, Dec 4, 2015 at 1:25 AM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Hi Bosco!
> When i run following command
> HADOOP_USER_NAME=mike hdfs dfs -ls /perm
>
> it is showing allowed because of hadoop_ACL, Why it is not because of
> ranger-acl?
>
> -12/04/2015 01:22:28 AMmike
> hadoopdev
> hdfs
> /permREAD_EXECUTEAllowedhadoop-acl192.168.23.1261
>
> On Fri, Dec 4, 2015 at 1:21 AM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
>>
>>
>> 12/04/2015 01:19:16 AMmike
>> hadoopdev
>> hdfs
>> /perm/m2WRITEDeniedhadoop-acl192.168.23.1261
>>
>> On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com>
>> wrote:
>>
>>> Yes, Bosco, it is denied , i made the mistake when i ran this test case
>>> i have changed mike permission to read and execute. Test_Id 7 is Denied
>>>
>>> Thanks
>>>
>>> On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> Can we check the audit log? If Ranger is giving the write permission,
>>>> then it is a bug.
>>>>
>>>> Thanks
>>>>
>>>> Bosco
>>>>
>>>>
>>>> From: Aneela Saleem <an...@platalytics.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Thursday, December 3, 2015 at 12:08 PM
>>>>
>>>> To: <us...@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Agreed with Bosco
>>>>
>>>> On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> Hafiz, thanks for testing it.
>>>>>
>>>>> Regard test case #7, shouldn't Mike been denied?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Thursday, December 3, 2015 at 11:54 AM
>>>>>
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Hi Bosco*,*
>>>>>
>>>>> I have tested Deny, Allow , Exclude from deny and exclude from Allow
>>>>> permissions. Following are details of my findings.
>>>>>
>>>>> *Test Cases*
>>>>>
>>>>> *Developer Group: *Roger, Smith
>>>>>
>>>>> *Data Scientist Group: *Mike, Clark
>>>>>
>>>>> Hadoop Resource : */perm*
>>>>> Ranger Policy
>>>>> [image: Inline image 2]
>>>>>
>>>>> Developers can do nothing having no read, write and execute
>>>>> permissions except Roger who has full permissions .All users of group
>>>>> Data-Scientist have permissions read, write and execute permissions
>>>>> except *mike* who can't write.
>>>>> Test Cases
>>>>>
>>>>> Test_ID
>>>>>
>>>>> User
>>>>>
>>>>> Group
>>>>>
>>>>> Command
>>>>>
>>>>> Expected
>>>>>
>>>>> Actual
>>>>>
>>>>> 1
>>>>>
>>>>> Roger
>>>>>
>>>>> Developers
>>>>>
>>>>> Hdfs dfs -ls /perm
>>>>>
>>>>> Allowed
>>>>>
>>>>> Allowed
>>>>>
>>>>> 2
>>>>>
>>>>> Roger
>>>>>
>>>>> Developers
>>>>>
>>>>> Hdfs dfs -mkdir /perm/r
>>>>>
>>>>> Allowed
>>>>>
>>>>> Allowed
>>>>>
>>>>> 3
>>>>>
>>>>> Smith
>>>>>
>>>>> Developers
>>>>>
>>>>> Hdfs dfs -mkdir /perm/s
>>>>>
>>>>> Denied
>>>>>
>>>>> Denied
>>>>>
>>>>> 4
>>>>>
>>>>> Smith
>>>>>
>>>>> Developers
>>>>>
>>>>> Hdfs dfs -ls /perm
>>>>>
>>>>> Denied
>>>>>
>>>>> Denied
>>>>>
>>>>> 5
>>>>>
>>>>> Clark
>>>>>
>>>>> Data-Scientis
>>>>>
>>>>> Hdfs dfs -ls /perm
>>>>>
>>>>> Allowed
>>>>>
>>>>> Allowed
>>>>>
>>>>> 6
>>>>>
>>>>> Clark
>>>>>
>>>>> Data-Scientis
>>>>>
>>>>> Hdfs dfs -mkdir /perm/c
>>>>>
>>>>> Allowed
>>>>>
>>>>> Allowed
>>>>>
>>>>> 7
>>>>>
>>>>> Mike
>>>>>
>>>>> Data-Scientis
>>>>>
>>>>> Hdfs dfs -mkdir /perm/m
>>>>>
>>>>> Alowed
>>>>>
>>>>> Allowed
>>>>>
>>>>> 8
>>>>>
>>>>> Mike
>>>>>
>>>>> Data-Scientis
>>>>>
>>>>> Hdfs dfs -ls /perm
>>>>> Allowed
>>>>>
>>>>> Allowed
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <
>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>
>>>>>> Hi Bosco,
>>>>>>
>>>>>> Thanks for your response, I am testing new feature of ranger
>>>>>> Deny,Allow. will send you my findings in short.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> >I want to know why audits are showing that it is because of
>>>>>>> hadoop-acl not ranger-acl?
>>>>>>> Hafiz, this is a good question and we should probably document it or
>>>>>>> come with a blog for this.
>>>>>>>
>>>>>>> Only for HDFS and YARN, we support falling back to native permission
>>>>>>> check if we don’t have corresponding permission in Ranger. So in your case,
>>>>>>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>>>>>>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>>>>>>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>>>>>>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>>>>>>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>>>>>>> creating the folder and the audit would should that hadoop-acl allowed to
>>>>>>> create the folder.
>>>>>>>
>>>>>>> This also answers yours previous question why we want to make
>>>>>>> umask=077 and chmod –r 000 to all application folders to be managed by
>>>>>>> Ranger. So if there are no Ranger policies, then we want to hadoop also to
>>>>>>> deny.
>>>>>>>
>>>>>>> With the recent deny feature, you can explicitly “deny” “asma” or
>>>>>>> any group from creating/writing. Or you could deny all, but exclude
>>>>>>> “developer’ and “sadaf” from the deny users.
>>>>>>>
>>>>>>> In the future release, I feel, we should provide a way to mark
>>>>>>> certain folders to be managed exclusively by Ranger. And that will remove a
>>>>>>> lot of confusion and also make the policy management more predictable.
>>>>>>>
>>>>>>> Does it answer your question?
>>>>>>>
>>>>>>> Bosco
>>>>>>>
>>>>>>>
>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>>>>>
>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Hi Bosco!
>>>>>>>
>>>>>>> I created a directory /mjd with following permissions
>>>>>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>>>>>
>>>>>>> Then i made a policy with following permissions
>>>>>>> [image: Inline image 1]
>>>>>>> Datascientist group has one user asma and developer group has one
>>>>>>> user named haniya and sadaf has no group.
>>>>>>>
>>>>>>> So when i run following command
>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>>>>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>>>>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *And audit of this command is as follow*ServicePolicy IDEvent Time
>>>>>>> UserName / TypeResource NameAccess TypeResultAccess EnforcerClient
>>>>>>> IPEvent Count--12/02/2015 09:46:23 AMasma
>>>>>>> hdfsRepo
>>>>>>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>>>>>>> I want to know why audits are showing that it is because of
>>>>>>> hadoop-acl not ranger-acl?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> You don’t need to. Since auditing is working, you can check who
>>>>>>>> gave the permission without 000
>>>>>>>>
>>>>>>>> We recommend giving 000 at HDFS level, because Ranger by default
>>>>>>>> falls back to HDFS permission. So for all folders you want to Ranger to be
>>>>>>>> exclusive, you give as minimal permission as possible.
>>>>>>>>
>>>>>>>> I think, we should also make it configurable in Ranger. Where you
>>>>>>>> can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you
>>>>>>>> don’t have to worry about HDFS level ACLs.
>>>>>>>>
>>>>>>>> The reason you don’t want Ranger to manage everything because there
>>>>>>>> are folders like tmp and user folders which want the system and user to
>>>>>>>> manage themselves. But for application folders like Hive warehouse, you
>>>>>>>> should let Ranger manage it.
>>>>>>>>
>>>>>>>> Bosco
>>>>>>>>
>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>>>>>
>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Bosco, I noticed group level permission works when we set hadoop
>>>>>>>> permissions to 000. I am just curious why it is so ?
>>>>>>>>
>>>>>>>> is it always necessary to set hadoop permissions to 000 for ranger
>>>>>>>> to work?
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>>>>>> related auditing is being shown
>>>>>>>>>
>>>>>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <
>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, you should fix audit first. That will help in debugging
>>>>>>>>>> these issues also.
>>>>>>>>>>
>>>>>>>>>> BTW, are you using Solr or DB?
>>>>>>>>>>
>>>>>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>>>>>> package for setting up Solr. It is available as attachment in
>>>>>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The
>>>>>>>>>> instructions are in
>>>>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>>>>>
>>>>>>>>>> Give it a try.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Bosco
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>>>>>
>>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>
>>>>>>>>>> Hafiz,
>>>>>>>>>>
>>>>>>>>>> Few things to check:
>>>>>>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>>>>>
>>>>>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>>>>>
>>>>>>>>>> Madhan
>>>>>>>>>>
>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>
>>>>>>>>>> Bosco,
>>>>>>>>>>
>>>>>>>>>> I have followed above steps
>>>>>>>>>>
>>>>>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49
>>>>>>>>>> /pg
>>>>>>>>>> 2. changed the umask so newly created folder or files have
>>>>>>>>>> following permissions
>>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>>>>>> hduser:hadoop
>>>>>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>>
>>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> thanks
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Bosco,
>>>>>>>>>>>
>>>>>>>>>>> I have followed above steps
>>>>>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>>>>> changed the umask so newly created folder or files have
>>>>>>>>>>> following permissions
>>>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>>>>>>
>>>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <
>>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you check Ranger Audits?
>>>>>>>>>>>>
>>>>>>>>>>>> Also, do couple of things:
>>>>>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>>>
>>>>>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>>>>>
>>>>>>>>>>>> Bosco
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>>>
>>>>>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <
>>>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bosco
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>>>>>> working
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> But when i run following command
>>>>>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>>>>>
>>>>>>>>>>>>> i works successfully
>>>>>>>>>>>>> what is the issue? ldap users and groups are synced correctly
>>>>>>>>>>>>> as when i run the command *hdfs groups asma* it returns
>>>>>>>>>>>>> correct group
>>>>>>>>>>>>> asma : datascientist
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Hi Bosco!
When i run following command
HADOOP_USER_NAME=mike hdfs dfs -ls /perm
it is showing allowed because of hadoop_ACL, Why it is not because of
ranger-acl?
-12/04/2015 01:22:28 AMmike
hadoopdev
hdfs
/permREAD_EXECUTEAllowedhadoop-acl192.168.23.1261
On Fri, Dec 4, 2015 at 1:21 AM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
>
>
> 12/04/2015 01:19:16 AMmike
> hadoopdev
> hdfs
> /perm/m2WRITEDeniedhadoop-acl192.168.23.1261
>
> On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Yes, Bosco, it is denied , i made the mistake when i ran this test case i
>> have changed mike permission to read and execute. Test_Id 7 is Denied
>>
>> Thanks
>>
>> On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>>> Can we check the audit log? If Ranger is giving the write permission,
>>> then it is a bug.
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
>>> From: Aneela Saleem <an...@platalytics.com>
>>> Reply-To: <us...@ranger.incubator.apache.org>
>>> Date: Thursday, December 3, 2015 at 12:08 PM
>>>
>>> To: <us...@ranger.incubator.apache.org>
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Agreed with Bosco
>>>
>>> On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> Hafiz, thanks for testing it.
>>>>
>>>> Regard test case #7, shouldn't Mike been denied?
>>>>
>>>> Thanks
>>>>
>>>> Bosco
>>>>
>>>>
>>>>
>>>>
>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Thursday, December 3, 2015 at 11:54 AM
>>>>
>>>> To: <us...@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Hi Bosco*,*
>>>>
>>>> I have tested Deny, Allow , Exclude from deny and exclude from Allow
>>>> permissions. Following are details of my findings.
>>>>
>>>> *Test Cases*
>>>>
>>>> *Developer Group: *Roger, Smith
>>>>
>>>> *Data Scientist Group: *Mike, Clark
>>>>
>>>> Hadoop Resource : */perm*
>>>> Ranger Policy
>>>> [image: Inline image 2]
>>>>
>>>> Developers can do nothing having no read, write and execute permissions
>>>> except Roger who has full permissions .All users of group Data-Scientist
>>>> have permissions read, write and execute permissions except *mike*
>>>> who can't write.
>>>> Test Cases
>>>>
>>>> Test_ID
>>>>
>>>> User
>>>>
>>>> Group
>>>>
>>>> Command
>>>>
>>>> Expected
>>>>
>>>> Actual
>>>>
>>>> 1
>>>>
>>>> Roger
>>>>
>>>> Developers
>>>>
>>>> Hdfs dfs -ls /perm
>>>>
>>>> Allowed
>>>>
>>>> Allowed
>>>>
>>>> 2
>>>>
>>>> Roger
>>>>
>>>> Developers
>>>>
>>>> Hdfs dfs -mkdir /perm/r
>>>>
>>>> Allowed
>>>>
>>>> Allowed
>>>>
>>>> 3
>>>>
>>>> Smith
>>>>
>>>> Developers
>>>>
>>>> Hdfs dfs -mkdir /perm/s
>>>>
>>>> Denied
>>>>
>>>> Denied
>>>>
>>>> 4
>>>>
>>>> Smith
>>>>
>>>> Developers
>>>>
>>>> Hdfs dfs -ls /perm
>>>>
>>>> Denied
>>>>
>>>> Denied
>>>>
>>>> 5
>>>>
>>>> Clark
>>>>
>>>> Data-Scientis
>>>>
>>>> Hdfs dfs -ls /perm
>>>>
>>>> Allowed
>>>>
>>>> Allowed
>>>>
>>>> 6
>>>>
>>>> Clark
>>>>
>>>> Data-Scientis
>>>>
>>>> Hdfs dfs -mkdir /perm/c
>>>>
>>>> Allowed
>>>>
>>>> Allowed
>>>>
>>>> 7
>>>>
>>>> Mike
>>>>
>>>> Data-Scientis
>>>>
>>>> Hdfs dfs -mkdir /perm/m
>>>>
>>>> Alowed
>>>>
>>>> Allowed
>>>>
>>>> 8
>>>>
>>>> Mike
>>>>
>>>> Data-Scientis
>>>>
>>>> Hdfs dfs -ls /perm
>>>> Allowed
>>>>
>>>> Allowed
>>>>
>>>>
>>>>
>>>> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <
>>>> hafizmujadid00@gmail.com> wrote:
>>>>
>>>>> Hi Bosco,
>>>>>
>>>>> Thanks for your response, I am testing new feature of ranger
>>>>> Deny,Allow. will send you my findings in short.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> >I want to know why audits are showing that it is because of
>>>>>> hadoop-acl not ranger-acl?
>>>>>> Hafiz, this is a good question and we should probably document it or
>>>>>> come with a blog for this.
>>>>>>
>>>>>> Only for HDFS and YARN, we support falling back to native permission
>>>>>> check if we don’t have corresponding permission in Ranger. So in your case,
>>>>>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>>>>>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>>>>>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>>>>>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>>>>>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>>>>>> creating the folder and the audit would should that hadoop-acl allowed to
>>>>>> create the folder.
>>>>>>
>>>>>> This also answers yours previous question why we want to make
>>>>>> umask=077 and chmod –r 000 to all application folders to be managed by
>>>>>> Ranger. So if there are no Ranger policies, then we want to hadoop also to
>>>>>> deny.
>>>>>>
>>>>>> With the recent deny feature, you can explicitly “deny” “asma” or any
>>>>>> group from creating/writing. Or you could deny all, but exclude “developer’
>>>>>> and “sadaf” from the deny users.
>>>>>>
>>>>>> In the future release, I feel, we should provide a way to mark
>>>>>> certain folders to be managed exclusively by Ranger. And that will remove a
>>>>>> lot of confusion and also make the policy management more predictable.
>>>>>>
>>>>>> Does it answer your question?
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>>>>
>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Hi Bosco!
>>>>>>
>>>>>> I created a directory /mjd with following permissions
>>>>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>>>>
>>>>>> Then i made a policy with following permissions
>>>>>> [image: Inline image 1]
>>>>>> Datascientist group has one user asma and developer group has one
>>>>>> user named haniya and sadaf has no group.
>>>>>>
>>>>>> So when i run following command
>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>>>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>>>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>>>>
>>>>>>
>>>>>>
>>>>>> *And audit of this command is as follow*ServicePolicy IDEvent Time
>>>>>> UserName / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent
>>>>>> Count--12/02/2015 09:46:23 AMasma
>>>>>> hdfsRepo
>>>>>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>>>>>> I want to know why audits are showing that it is because of
>>>>>> hadoop-acl not ranger-acl?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> You don’t need to. Since auditing is working, you can check who gave
>>>>>>> the permission without 000
>>>>>>>
>>>>>>> We recommend giving 000 at HDFS level, because Ranger by default
>>>>>>> falls back to HDFS permission. So for all folders you want to Ranger to be
>>>>>>> exclusive, you give as minimal permission as possible.
>>>>>>>
>>>>>>> I think, we should also make it configurable in Ranger. Where you
>>>>>>> can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you
>>>>>>> don’t have to worry about HDFS level ACLs.
>>>>>>>
>>>>>>> The reason you don’t want Ranger to manage everything because there
>>>>>>> are folders like tmp and user folders which want the system and user to
>>>>>>> manage themselves. But for application folders like Hive warehouse, you
>>>>>>> should let Ranger manage it.
>>>>>>>
>>>>>>> Bosco
>>>>>>>
>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>>>>
>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Bosco, I noticed group level permission works when we set hadoop
>>>>>>> permissions to 000. I am just curious why it is so ?
>>>>>>>
>>>>>>> is it always necessary to set hadoop permissions to 000 for ranger
>>>>>>> to work?
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>
>>>>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>>>>> related auditing is being shown
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bosco@apache.org
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Yes, you should fix audit first. That will help in debugging these
>>>>>>>>> issues also.
>>>>>>>>>
>>>>>>>>> BTW, are you using Solr or DB?
>>>>>>>>>
>>>>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>>>>> package for setting up Solr. It is available as attachment in
>>>>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The
>>>>>>>>> instructions are in
>>>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>>>>
>>>>>>>>> Give it a try.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Bosco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>>>>
>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>
>>>>>>>>> Hafiz,
>>>>>>>>>
>>>>>>>>> Few things to check:
>>>>>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>>>>
>>>>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>>>>
>>>>>>>>> Madhan
>>>>>>>>>
>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>
>>>>>>>>> Bosco,
>>>>>>>>>
>>>>>>>>> I have followed above steps
>>>>>>>>>
>>>>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>>> 2. changed the umask so newly created folder or files have
>>>>>>>>> following permissions
>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>>>>> hduser:hadoop
>>>>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>
>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>>
>>>>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Bosco,
>>>>>>>>>>
>>>>>>>>>> I have followed above steps
>>>>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>>>> changed the umask so newly created folder or files have following
>>>>>>>>>> permissions
>>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>>>>>
>>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <
>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can you check Ranger Audits?
>>>>>>>>>>>
>>>>>>>>>>> Also, do couple of things:
>>>>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>>
>>>>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>>>>
>>>>>>>>>>> Bosco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>>
>>>>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <
>>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Bosco
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>>>>
>>>>>>>>>>>> Hi all
>>>>>>>>>>>>
>>>>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>>>>> working
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But when i run following command
>>>>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>>>>
>>>>>>>>>>>> i works successfully
>>>>>>>>>>>> what is the issue? ldap users and groups are synced correctly
>>>>>>>>>>>> as when i run the command *hdfs groups asma* it returns
>>>>>>>>>>>> correct group
>>>>>>>>>>>> asma : datascientist
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Don Bosco Durai <bo...@apache.org>.
That is good news.
I like your representation of the use cases and it’s test results. If I give write access to Ranger wiki, would you be able to put this (with the images) in wiki. I think, it will be very useful for everyone.
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 12:21 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
12/04/2015 01:19:16 AMmikehadoopdev
hdfs
/perm/m2WRITEDeniedhadoop-acl192.168.23.1261
On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com> wrote:
Yes, Bosco, it is denied , i made the mistake when i ran this test case i have changed mike permission to read and execute. Test_Id 7 is Denied
Thanks
On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can we check the audit log? If Ranger is giving the write permission, then it is a bug.
Thanks
Bosco
From: Aneela Saleem <an...@platalytics.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 12:08 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Agreed with Bosco
On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
Hafiz, thanks for testing it.
Regard test case #7, shouldn't Mike been denied?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 11:54 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco,
I have tested Deny, Allow , Exclude from deny and exclude from Allow permissions. Following are details of my findings.
Test Cases
Developer Group: Roger, Smith
Data Scientist Group: Mike, Clark
Hadoop Resource : /perm
Ranger Policy
Developers can do nothing having no read, write and execute permissions except Roger who has full permissions .All users of group Data-Scientist have permissions read, write and execute permissions except mike who can't write.
Test Cases
Test_ID User Group Command Expected Actual
1 Roger Developers Hdfs dfs -ls /perm Allowed Allowed
2 Roger Developers Hdfs dfs -mkdir /perm/r Allowed Allowed
3 Smith Developers Hdfs dfs -mkdir /perm/s Denied Denied
4 Smith Developers Hdfs dfs -ls /perm Denied Denied
5 Clark Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
6 Clark Data-Scientis Hdfs dfs -mkdir /perm/c Allowed Allowed
7 Mike Data-Scientis Hdfs dfs -mkdir /perm/m Alowed Allowed
8 Mike Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Hi Bosco,
Thanks for your response, I am testing new feature of ranger Deny,Allow. will send you my findings in short.
Thanks
On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
>I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Hafiz, this is a good question and we should probably document it or come with a blog for this.
Only for HDFS and YARN, we support falling back to native permission check if we don’t have corresponding permission in Ranger. So in your case, since there were no permissions in Ranger for “asma” to the folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t have native posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed creating the folder and the audit would should that hadoop-acl allowed to create the folder.
This also answers yours previous question why we want to make umask=077 and chmod –r 000 to all application folders to be managed by Ranger. So if there are no Ranger policies, then we want to hadoop also to deny.
With the recent deny feature, you can explicitly “deny” “asma” or any group from creating/writing. Or you could deny all, but exclude “developer’ and “sadaf” from the deny users.
In the future release, I feel, we should provide a way to mark certain folders to be managed exclusively by Ranger. And that will remove a lot of confusion and also make the policy management more predictable.
Does it answer your question?
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 8:59 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco!
I created a directory /mjd with following permissions
drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd
Then i made a policy with following permissions
Datascientist group has one user asma and developer group has one user named haniya and sadaf has no group.
So when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1
mkdir: Permission denied: user=asma, access=WRITE, inode="/mjd/a1":hduser:supergroup:drwxr-xr-x
And audit of this command is as follow
Service
Policy IDEvent TimeUserName / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
--12/02/2015 09:46:23 AMasmahdfsRepo
/mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Audit for HADOOP_USER_NAME=mike hdfs dfs -mkdir /perm/m2
12/04/2015 01:19:16 AMmike
hadoopdev
hdfs
/perm/m2WRITEDeniedhadoop-acl192.168.23.1261
On Fri, Dec 4, 2015 at 1:20 AM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Yes, Bosco, it is denied , i made the mistake when i ran this test case i
> have changed mike permission to read and execute. Test_Id 7 is Denied
>
> Thanks
>
> On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org> wrote:
>
>> Can we check the audit log? If Ranger is giving the write permission,
>> then it is a bug.
>>
>> Thanks
>>
>> Bosco
>>
>>
>> From: Aneela Saleem <an...@platalytics.com>
>> Reply-To: <us...@ranger.incubator.apache.org>
>> Date: Thursday, December 3, 2015 at 12:08 PM
>>
>> To: <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Agreed with Bosco
>>
>> On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>>> Hafiz, thanks for testing it.
>>>
>>> Regard test case #7, shouldn't Mike been denied?
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
>>>
>>>
>>> From: Hafiz Mujadid <ha...@gmail.com>
>>> Reply-To: <us...@ranger.incubator.apache.org>
>>> Date: Thursday, December 3, 2015 at 11:54 AM
>>>
>>> To: <us...@ranger.incubator.apache.org>
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Hi Bosco*,*
>>>
>>> I have tested Deny, Allow , Exclude from deny and exclude from Allow
>>> permissions. Following are details of my findings.
>>>
>>> *Test Cases*
>>>
>>> *Developer Group: *Roger, Smith
>>>
>>> *Data Scientist Group: *Mike, Clark
>>>
>>> Hadoop Resource : */perm*
>>> Ranger Policy
>>> [image: Inline image 2]
>>>
>>> Developers can do nothing having no read, write and execute permissions
>>> except Roger who has full permissions .All users of group Data-Scientist
>>> have permissions read, write and execute permissions except *mike*
>>> who can't write.
>>> Test Cases
>>>
>>> Test_ID
>>>
>>> User
>>>
>>> Group
>>>
>>> Command
>>>
>>> Expected
>>>
>>> Actual
>>>
>>> 1
>>>
>>> Roger
>>>
>>> Developers
>>>
>>> Hdfs dfs -ls /perm
>>>
>>> Allowed
>>>
>>> Allowed
>>>
>>> 2
>>>
>>> Roger
>>>
>>> Developers
>>>
>>> Hdfs dfs -mkdir /perm/r
>>>
>>> Allowed
>>>
>>> Allowed
>>>
>>> 3
>>>
>>> Smith
>>>
>>> Developers
>>>
>>> Hdfs dfs -mkdir /perm/s
>>>
>>> Denied
>>>
>>> Denied
>>>
>>> 4
>>>
>>> Smith
>>>
>>> Developers
>>>
>>> Hdfs dfs -ls /perm
>>>
>>> Denied
>>>
>>> Denied
>>>
>>> 5
>>>
>>> Clark
>>>
>>> Data-Scientis
>>>
>>> Hdfs dfs -ls /perm
>>>
>>> Allowed
>>>
>>> Allowed
>>>
>>> 6
>>>
>>> Clark
>>>
>>> Data-Scientis
>>>
>>> Hdfs dfs -mkdir /perm/c
>>>
>>> Allowed
>>>
>>> Allowed
>>>
>>> 7
>>>
>>> Mike
>>>
>>> Data-Scientis
>>>
>>> Hdfs dfs -mkdir /perm/m
>>>
>>> Alowed
>>>
>>> Allowed
>>>
>>> 8
>>>
>>> Mike
>>>
>>> Data-Scientis
>>>
>>> Hdfs dfs -ls /perm
>>> Allowed
>>>
>>> Allowed
>>>
>>>
>>>
>>> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <hafizmujadid00@gmail.com
>>> > wrote:
>>>
>>>> Hi Bosco,
>>>>
>>>> Thanks for your response, I am testing new feature of ranger
>>>> Deny,Allow. will send you my findings in short.
>>>>
>>>> Thanks
>>>>
>>>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> >I want to know why audits are showing that it is because of
>>>>> hadoop-acl not ranger-acl?
>>>>> Hafiz, this is a good question and we should probably document it or
>>>>> come with a blog for this.
>>>>>
>>>>> Only for HDFS and YARN, we support falling back to native permission
>>>>> check if we don’t have corresponding permission in Ranger. So in your case,
>>>>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>>>>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>>>>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>>>>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>>>>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>>>>> creating the folder and the audit would should that hadoop-acl allowed to
>>>>> create the folder.
>>>>>
>>>>> This also answers yours previous question why we want to make
>>>>> umask=077 and chmod –r 000 to all application folders to be managed by
>>>>> Ranger. So if there are no Ranger policies, then we want to hadoop also to
>>>>> deny.
>>>>>
>>>>> With the recent deny feature, you can explicitly “deny” “asma” or any
>>>>> group from creating/writing. Or you could deny all, but exclude “developer’
>>>>> and “sadaf” from the deny users.
>>>>>
>>>>> In the future release, I feel, we should provide a way to mark certain
>>>>> folders to be managed exclusively by Ranger. And that will remove a lot of
>>>>> confusion and also make the policy management more predictable.
>>>>>
>>>>> Does it answer your question?
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>>>
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Hi Bosco!
>>>>>
>>>>> I created a directory /mjd with following permissions
>>>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>>>
>>>>> Then i made a policy with following permissions
>>>>> [image: Inline image 1]
>>>>> Datascientist group has one user asma and developer group has one user
>>>>> named haniya and sadaf has no group.
>>>>>
>>>>> So when i run following command
>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>>>
>>>>>
>>>>>
>>>>> *And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
>>>>> / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent
>>>>> Count--12/02/2015 09:46:23 AMasma
>>>>> hdfsRepo
>>>>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>>>>> I want to know why audits are showing that it is because of hadoop-acl
>>>>> not ranger-acl?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> You don’t need to. Since auditing is working, you can check who gave
>>>>>> the permission without 000
>>>>>>
>>>>>> We recommend giving 000 at HDFS level, because Ranger by default
>>>>>> falls back to HDFS permission. So for all folders you want to Ranger to be
>>>>>> exclusive, you give as minimal permission as possible.
>>>>>>
>>>>>> I think, we should also make it configurable in Ranger. Where you can
>>>>>> tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t
>>>>>> have to worry about HDFS level ACLs.
>>>>>>
>>>>>> The reason you don’t want Ranger to manage everything because there
>>>>>> are folders like tmp and user folders which want the system and user to
>>>>>> manage themselves. But for application folders like Hive warehouse, you
>>>>>> should let Ranger manage it.
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>>>
>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Bosco, I noticed group level permission works when we set hadoop
>>>>>> permissions to 000. I am just curious why it is so ?
>>>>>>
>>>>>> is it always necessary to set hadoop permissions to 000 for ranger to
>>>>>> work?
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>
>>>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>>>> related auditing is being shown
>>>>>>>
>>>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, you should fix audit first. That will help in debugging these
>>>>>>>> issues also.
>>>>>>>>
>>>>>>>> BTW, are you using Solr or DB?
>>>>>>>>
>>>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>>>> package for setting up Solr. It is available as attachment in
>>>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions
>>>>>>>> are in
>>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>>>
>>>>>>>> Give it a try.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Bosco
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>>>
>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Hafiz,
>>>>>>>>
>>>>>>>> Few things to check:
>>>>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>>>
>>>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>>>
>>>>>>>> Madhan
>>>>>>>>
>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>>> user@ranger.incubator.apache.org>
>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Bosco,
>>>>>>>>
>>>>>>>> I have followed above steps
>>>>>>>>
>>>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>> 2. changed the umask so newly created folder or files have
>>>>>>>> following permissions
>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>>>> hduser:hadoop
>>>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>>>
>>>>>>>>
>>>>>>>> but still group level permissions are not working.
>>>>>>>>
>>>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>>>> audits. i will let you know when audits are available.
>>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Bosco,
>>>>>>>>>
>>>>>>>>> I have followed above steps
>>>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>>> changed the umask so newly created folder or files have following
>>>>>>>>> permissions
>>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>>>>
>>>>>>>>> but still group level permissions are not working.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> my audits are not working, i am trying to figure out the issue
>>>>>>>>> with audits. i will let you know when audits are available.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bosco@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Can you check Ranger Audits?
>>>>>>>>>>
>>>>>>>>>> Also, do couple of things:
>>>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>>>
>>>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>>>
>>>>>>>>>> Bosco
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>>
>>>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <
>>>>>>>>>> bosco@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Bosco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>>>
>>>>>>>>>>> Hi all
>>>>>>>>>>>
>>>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>>>> working
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But when i run following command
>>>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>>>
>>>>>>>>>>> i works successfully
>>>>>>>>>>> what is the issue? ldap users and groups are synced correctly as
>>>>>>>>>>> when i run the command *hdfs groups asma* it returns correct
>>>>>>>>>>> group
>>>>>>>>>>> asma : datascientist
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>>
>>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Yes, Bosco, it is denied , i made the mistake when i ran this test case i
have changed mike permission to read and execute. Test_Id 7 is Denied
Thanks
On Fri, Dec 4, 2015 at 1:10 AM, Don Bosco Durai <bo...@apache.org> wrote:
> Can we check the audit log? If Ranger is giving the write permission, then
> it is a bug.
>
> Thanks
>
> Bosco
>
>
> From: Aneela Saleem <an...@platalytics.com>
> Reply-To: <us...@ranger.incubator.apache.org>
> Date: Thursday, December 3, 2015 at 12:08 PM
>
> To: <us...@ranger.incubator.apache.org>
> Subject: Re: Group level permission are not working in ranger
>
> Agreed with Bosco
>
> On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
>
>> Hafiz, thanks for testing it.
>>
>> Regard test case #7, shouldn't Mike been denied?
>>
>> Thanks
>>
>> Bosco
>>
>>
>>
>>
>> From: Hafiz Mujadid <ha...@gmail.com>
>> Reply-To: <us...@ranger.incubator.apache.org>
>> Date: Thursday, December 3, 2015 at 11:54 AM
>>
>> To: <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Hi Bosco*,*
>>
>> I have tested Deny, Allow , Exclude from deny and exclude from Allow
>> permissions. Following are details of my findings.
>>
>> *Test Cases*
>>
>> *Developer Group: *Roger, Smith
>>
>> *Data Scientist Group: *Mike, Clark
>>
>> Hadoop Resource : */perm*
>> Ranger Policy
>> [image: Inline image 2]
>>
>> Developers can do nothing having no read, write and execute permissions
>> except Roger who has full permissions .All users of group Data-Scientist
>> have permissions read, write and execute permissions except *mike* who
>> can't write.
>> Test Cases
>>
>> Test_ID
>>
>> User
>>
>> Group
>>
>> Command
>>
>> Expected
>>
>> Actual
>>
>> 1
>>
>> Roger
>>
>> Developers
>>
>> Hdfs dfs -ls /perm
>>
>> Allowed
>>
>> Allowed
>>
>> 2
>>
>> Roger
>>
>> Developers
>>
>> Hdfs dfs -mkdir /perm/r
>>
>> Allowed
>>
>> Allowed
>>
>> 3
>>
>> Smith
>>
>> Developers
>>
>> Hdfs dfs -mkdir /perm/s
>>
>> Denied
>>
>> Denied
>>
>> 4
>>
>> Smith
>>
>> Developers
>>
>> Hdfs dfs -ls /perm
>>
>> Denied
>>
>> Denied
>>
>> 5
>>
>> Clark
>>
>> Data-Scientis
>>
>> Hdfs dfs -ls /perm
>>
>> Allowed
>>
>> Allowed
>>
>> 6
>>
>> Clark
>>
>> Data-Scientis
>>
>> Hdfs dfs -mkdir /perm/c
>>
>> Allowed
>>
>> Allowed
>>
>> 7
>>
>> Mike
>>
>> Data-Scientis
>>
>> Hdfs dfs -mkdir /perm/m
>>
>> Alowed
>>
>> Allowed
>>
>> 8
>>
>> Mike
>>
>> Data-Scientis
>>
>> Hdfs dfs -ls /perm
>> Allowed
>>
>> Allowed
>>
>>
>>
>> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com>
>> wrote:
>>
>>> Hi Bosco,
>>>
>>> Thanks for your response, I am testing new feature of ranger Deny,Allow.
>>> will send you my findings in short.
>>>
>>> Thanks
>>>
>>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> >I want to know why audits are showing that it is because of hadoop-acl
>>>> not ranger-acl?
>>>> Hafiz, this is a good question and we should probably document it or
>>>> come with a blog for this.
>>>>
>>>> Only for HDFS and YARN, we support falling back to native permission
>>>> check if we don’t have corresponding permission in Ranger. So in your case,
>>>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>>>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>>>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>>>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>>>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>>>> creating the folder and the audit would should that hadoop-acl allowed to
>>>> create the folder.
>>>>
>>>> This also answers yours previous question why we want to make umask=077
>>>> and chmod –r 000 to all application folders to be managed by Ranger. So if
>>>> there are no Ranger policies, then we want to hadoop also to deny.
>>>>
>>>> With the recent deny feature, you can explicitly “deny” “asma” or any
>>>> group from creating/writing. Or you could deny all, but exclude “developer’
>>>> and “sadaf” from the deny users.
>>>>
>>>> In the future release, I feel, we should provide a way to mark certain
>>>> folders to be managed exclusively by Ranger. And that will remove a lot of
>>>> confusion and also make the policy management more predictable.
>>>>
>>>> Does it answer your question?
>>>>
>>>> Bosco
>>>>
>>>>
>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>>
>>>> To: <us...@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Hi Bosco!
>>>>
>>>> I created a directory /mjd with following permissions
>>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>>
>>>> Then i made a policy with following permissions
>>>> [image: Inline image 1]
>>>> Datascientist group has one user asma and developer group has one user
>>>> named haniya and sadaf has no group.
>>>>
>>>> So when i run following command
>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>>
>>>>
>>>>
>>>> *And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
>>>> / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
>>>> --12/02/2015 09:46:23 AMasma
>>>> hdfsRepo
>>>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>>>> I want to know why audits are showing that it is because of hadoop-acl
>>>> not ranger-acl?
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> You don’t need to. Since auditing is working, you can check who gave
>>>>> the permission without 000
>>>>>
>>>>> We recommend giving 000 at HDFS level, because Ranger by default falls
>>>>> back to HDFS permission. So for all folders you want to Ranger to be
>>>>> exclusive, you give as minimal permission as possible.
>>>>>
>>>>> I think, we should also make it configurable in Ranger. Where you can
>>>>> tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t
>>>>> have to worry about HDFS level ACLs.
>>>>>
>>>>> The reason you don’t want Ranger to manage everything because there
>>>>> are folders like tmp and user folders which want the system and user to
>>>>> manage themselves. But for application folders like Hive warehouse, you
>>>>> should let Ranger manage it.
>>>>>
>>>>> Bosco
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>>
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Hi,
>>>>>
>>>>> Bosco, I noticed group level permission works when we set hadoop
>>>>> permissions to 000. I am just curious why it is so ?
>>>>>
>>>>> is it always necessary to set hadoop permissions to 000 for ranger to
>>>>> work?
>>>>>
>>>>> thanks
>>>>>
>>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>
>>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>>> related auditing is being shown
>>>>>>
>>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, you should fix audit first. That will help in debugging these
>>>>>>> issues also.
>>>>>>>
>>>>>>> BTW, are you using Solr or DB?
>>>>>>>
>>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>>> package for setting up Solr. It is available as attachment in
>>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions
>>>>>>> are in
>>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>>
>>>>>>> Give it a try.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Bosco
>>>>>>>
>>>>>>>
>>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>>
>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>> user@ranger.incubator.apache.org>
>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Hafiz,
>>>>>>>
>>>>>>> Few things to check:
>>>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>>
>>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>>
>>>>>>> Madhan
>>>>>>>
>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>>> user@ranger.incubator.apache.org>
>>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>>> user@ranger.incubator.apache.org>
>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Bosco,
>>>>>>>
>>>>>>> I have followed above steps
>>>>>>>
>>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>> 2. changed the umask so newly created folder or files have
>>>>>>> following permissions
>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>>> hduser:hadoop
>>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>>
>>>>>>>
>>>>>>> but still group level permissions are not working.
>>>>>>>
>>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>>> audits. i will let you know when audits are available.
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>>
>>>>>>>> Bosco,
>>>>>>>>
>>>>>>>> I have followed above steps
>>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>>> changed the umask so newly created folder or files have following
>>>>>>>> permissions
>>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>>>
>>>>>>>> but still group level permissions are not working.
>>>>>>>>
>>>>>>>>
>>>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>>>> audits. i will let you know when audits are available.
>>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Can you check Ranger Audits?
>>>>>>>>>
>>>>>>>>> Also, do couple of things:
>>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>>
>>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>>
>>>>>>>>> Bosco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>>
>>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>>
>>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bosco@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Bosco
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>>
>>>>>>>>>> Hi all
>>>>>>>>>>
>>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>>> working
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But when i run following command
>>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>>
>>>>>>>>>> i works successfully
>>>>>>>>>> what is the issue? ldap users and groups are synced correctly as
>>>>>>>>>> when i run the command *hdfs groups asma* it returns correct
>>>>>>>>>> group
>>>>>>>>>> asma : datascientist
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>>
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Don Bosco Durai <bo...@apache.org>.
Can we check the audit log? If Ranger is giving the write permission, then it is a bug.
Thanks
Bosco
From: Aneela Saleem <an...@platalytics.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 12:08 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Agreed with Bosco
On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
Hafiz, thanks for testing it.
Regard test case #7, shouldn't Mike been denied?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 11:54 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco,
I have tested Deny, Allow , Exclude from deny and exclude from Allow permissions. Following are details of my findings.
Test Cases
Developer Group: Roger, Smith
Data Scientist Group: Mike, Clark
Hadoop Resource : /perm
Ranger Policy
Developers can do nothing having no read, write and execute permissions except Roger who has full permissions .All users of group Data-Scientist have permissions read, write and execute permissions except mike who can't write.
Test Cases
Test_ID User Group Command Expected Actual
1 Roger Developers Hdfs dfs -ls /perm Allowed Allowed
2 Roger Developers Hdfs dfs -mkdir /perm/r Allowed Allowed
3 Smith Developers Hdfs dfs -mkdir /perm/s Denied Denied
4 Smith Developers Hdfs dfs -ls /perm Denied Denied
5 Clark Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
6 Clark Data-Scientis Hdfs dfs -mkdir /perm/c Allowed Allowed
7 Mike Data-Scientis Hdfs dfs -mkdir /perm/m Alowed Allowed
8 Mike Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Hi Bosco,
Thanks for your response, I am testing new feature of ranger Deny,Allow. will send you my findings in short.
Thanks
On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
>I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Hafiz, this is a good question and we should probably document it or come with a blog for this.
Only for HDFS and YARN, we support falling back to native permission check if we don’t have corresponding permission in Ranger. So in your case, since there were no permissions in Ranger for “asma” to the folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t have native posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed creating the folder and the audit would should that hadoop-acl allowed to create the folder.
This also answers yours previous question why we want to make umask=077 and chmod –r 000 to all application folders to be managed by Ranger. So if there are no Ranger policies, then we want to hadoop also to deny.
With the recent deny feature, you can explicitly “deny” “asma” or any group from creating/writing. Or you could deny all, but exclude “developer’ and “sadaf” from the deny users.
In the future release, I feel, we should provide a way to mark certain folders to be managed exclusively by Ranger. And that will remove a lot of confusion and also make the policy management more predictable.
Does it answer your question?
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 8:59 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco!
I created a directory /mjd with following permissions
drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd
Then i made a policy with following permissions
Datascientist group has one user asma and developer group has one user named haniya and sadaf has no group.
So when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1
mkdir: Permission denied: user=asma, access=WRITE, inode="/mjd/a1":hduser:supergroup:drwxr-xr-x
And audit of this command is as follow
Service
Policy IDEvent TimeUserName / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
--12/02/2015 09:46:23 AMasmahdfsRepo
/mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Aneela Saleem <an...@platalytics.com>.
Agreed with Bosco
On Fri, Dec 4, 2015 at 1:00 AM, Don Bosco Durai <bo...@apache.org> wrote:
> Hafiz, thanks for testing it.
>
> Regard test case #7, shouldn't Mike been denied?
>
> Thanks
>
> Bosco
>
>
>
>
> From: Hafiz Mujadid <ha...@gmail.com>
> Reply-To: <us...@ranger.incubator.apache.org>
> Date: Thursday, December 3, 2015 at 11:54 AM
>
> To: <us...@ranger.incubator.apache.org>
> Subject: Re: Group level permission are not working in ranger
>
> Hi Bosco*,*
>
> I have tested Deny, Allow , Exclude from deny and exclude from Allow
> permissions. Following are details of my findings.
>
> *Test Cases*
>
> *Developer Group: *Roger, Smith
>
> *Data Scientist Group: *Mike, Clark
>
> Hadoop Resource : */perm*
> Ranger Policy
> [image: Inline image 2]
>
> Developers can do nothing having no read, write and execute permissions
> except Roger who has full permissions .All users of group Data-Scientist
> have permissions read, write and execute permissions except *mike* who
> can't write.
> Test Cases
>
> Test_ID
>
> User
>
> Group
>
> Command
>
> Expected
>
> Actual
>
> 1
>
> Roger
>
> Developers
>
> Hdfs dfs -ls /perm
>
> Allowed
>
> Allowed
>
> 2
>
> Roger
>
> Developers
>
> Hdfs dfs -mkdir /perm/r
>
> Allowed
>
> Allowed
>
> 3
>
> Smith
>
> Developers
>
> Hdfs dfs -mkdir /perm/s
>
> Denied
>
> Denied
>
> 4
>
> Smith
>
> Developers
>
> Hdfs dfs -ls /perm
>
> Denied
>
> Denied
>
> 5
>
> Clark
>
> Data-Scientis
>
> Hdfs dfs -ls /perm
>
> Allowed
>
> Allowed
>
> 6
>
> Clark
>
> Data-Scientis
>
> Hdfs dfs -mkdir /perm/c
>
> Allowed
>
> Allowed
>
> 7
>
> Mike
>
> Data-Scientis
>
> Hdfs dfs -mkdir /perm/m
>
> Alowed
>
> Allowed
>
> 8
>
> Mike
>
> Data-Scientis
>
> Hdfs dfs -ls /perm
> Allowed
>
> Allowed
>
>
>
> On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Hi Bosco,
>>
>> Thanks for your response, I am testing new feature of ranger Deny,Allow.
>> will send you my findings in short.
>>
>> Thanks
>>
>> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org>
>> wrote:
>>
>>> >I want to know why audits are showing that it is because of hadoop-acl
>>> not ranger-acl?
>>> Hafiz, this is a good question and we should probably document it or
>>> come with a blog for this.
>>>
>>> Only for HDFS and YARN, we support falling back to native permission
>>> check if we don’t have corresponding permission in Ranger. So in your case,
>>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>>> creating the folder and the audit would should that hadoop-acl allowed to
>>> create the folder.
>>>
>>> This also answers yours previous question why we want to make umask=077
>>> and chmod –r 000 to all application folders to be managed by Ranger. So if
>>> there are no Ranger policies, then we want to hadoop also to deny.
>>>
>>> With the recent deny feature, you can explicitly “deny” “asma” or any
>>> group from creating/writing. Or you could deny all, but exclude “developer’
>>> and “sadaf” from the deny users.
>>>
>>> In the future release, I feel, we should provide a way to mark certain
>>> folders to be managed exclusively by Ranger. And that will remove a lot of
>>> confusion and also make the policy management more predictable.
>>>
>>> Does it answer your question?
>>>
>>> Bosco
>>>
>>>
>>> From: Hafiz Mujadid <ha...@gmail.com>
>>> Reply-To: <us...@ranger.incubator.apache.org>
>>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>>
>>> To: <us...@ranger.incubator.apache.org>
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Hi Bosco!
>>>
>>> I created a directory /mjd with following permissions
>>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>>
>>> Then i made a policy with following permissions
>>> [image: Inline image 1]
>>> Datascientist group has one user asma and developer group has one user
>>> named haniya and sadaf has no group.
>>>
>>> So when i run following command
>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>>> *mkdir: Permission denied: user=asma, access=WRITE,
>>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>>
>>>
>>>
>>> *And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
>>> / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
>>> --12/02/2015 09:46:23 AMasma
>>> hdfsRepo
>>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>>> I want to know why audits are showing that it is because of hadoop-acl
>>> not ranger-acl?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> You don’t need to. Since auditing is working, you can check who gave
>>>> the permission without 000
>>>>
>>>> We recommend giving 000 at HDFS level, because Ranger by default falls
>>>> back to HDFS permission. So for all folders you want to Ranger to be
>>>> exclusive, you give as minimal permission as possible.
>>>>
>>>> I think, we should also make it configurable in Ranger. Where you can
>>>> tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t
>>>> have to worry about HDFS level ACLs.
>>>>
>>>> The reason you don’t want Ranger to manage everything because there are
>>>> folders like tmp and user folders which want the system and user to manage
>>>> themselves. But for application folders like Hive warehouse, you should let
>>>> Ranger manage it.
>>>>
>>>> Bosco
>>>>
>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>>
>>>> To: <us...@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Hi,
>>>>
>>>> Bosco, I noticed group level permission works when we set hadoop
>>>> permissions to 000. I am just curious why it is so ?
>>>>
>>>> is it always necessary to set hadoop permissions to 000 for ranger to
>>>> work?
>>>>
>>>> thanks
>>>>
>>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>>> hafizmujadid00@gmail.com> wrote:
>>>>
>>>>> Bosco, I have tried both mysql db and solr as well, only plugin
>>>>> related auditing is being shown
>>>>>
>>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Yes, you should fix audit first. That will help in debugging these
>>>>>> issues also.
>>>>>>
>>>>>> BTW, are you using Solr or DB?
>>>>>>
>>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>>> package for setting up Solr. It is available as attachment in
>>>>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions
>>>>>> are in
>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>>
>>>>>> Give it a try.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>>
>>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>>
>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>> user@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Hafiz,
>>>>>>
>>>>>> Few things to check:
>>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>>> 2. Can you disable this policy and try mkdir?
>>>>>>
>>>>>> Fixing the issue with audit will help; audit log will have the
>>>>>> details of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>>
>>>>>> Madhan
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>>> user@ranger.incubator.apache.org>
>>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>>> To: "user@ranger.incubator.apache.org" <
>>>>>> user@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Bosco,
>>>>>>
>>>>>> I have followed above steps
>>>>>>
>>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>> 2. changed the umask so newly created folder or files have
>>>>>> following permissions
>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>>> hduser:hadoop
>>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>>
>>>>>>
>>>>>> but still group level permissions are not working.
>>>>>>
>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>> audits. i will let you know when audits are available.
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>>
>>>>>>> Bosco,
>>>>>>>
>>>>>>> I have followed above steps
>>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>>> changed the umask so newly created folder or files have following
>>>>>>> permissions
>>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>>
>>>>>>> but still group level permissions are not working.
>>>>>>>
>>>>>>>
>>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>>> audits. i will let you know when audits are available.
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can you check Ranger Audits?
>>>>>>>>
>>>>>>>> Also, do couple of things:
>>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>>
>>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>>
>>>>>>>> Bosco
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Yes Bosco, directory is being created.
>>>>>>>>
>>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Bosco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>>
>>>>>>>>> Hi all
>>>>>>>>>
>>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>>> working
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But when i run following command
>>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>>
>>>>>>>>> i works successfully
>>>>>>>>> what is the issue? ldap users and groups are synced correctly as
>>>>>>>>> when i run the command *hdfs groups asma* it returns correct
>>>>>>>>> group
>>>>>>>>> asma : datascientist
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
>
Re: Group level permission are not working in ranger
Posted by Don Bosco Durai <bo...@apache.org>.
Hafiz, thanks for testing it.
Regard test case #7, shouldn't Mike been denied?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Thursday, December 3, 2015 at 11:54 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco,
I have tested Deny, Allow , Exclude from deny and exclude from Allow permissions. Following are details of my findings.
Test Cases
Developer Group: Roger, Smith
Data Scientist Group: Mike, Clark
Hadoop Resource : /perm
Ranger Policy
Developers can do nothing having no read, write and execute permissions except Roger who has full permissions .All users of group Data-Scientist have permissions read, write and execute permissions except mike who can't write.
Test Cases
Test_ID User Group Command Expected Actual
1 Roger Developers Hdfs dfs -ls /perm Allowed Allowed
2 Roger Developers Hdfs dfs -mkdir /perm/r Allowed Allowed
3 Smith Developers Hdfs dfs -mkdir /perm/s Denied Denied
4 Smith Developers Hdfs dfs -ls /perm Denied Denied
5 Clark Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
6 Clark Data-Scientis Hdfs dfs -mkdir /perm/c Allowed Allowed
7 Mike Data-Scientis Hdfs dfs -mkdir /perm/m Alowed Allowed
8 Mike Data-Scientis Hdfs dfs -ls /perm Allowed Allowed
On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Hi Bosco,
Thanks for your response, I am testing new feature of ranger Deny,Allow. will send you my findings in short.
Thanks
On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
>I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Hafiz, this is a good question and we should probably document it or come with a blog for this.
Only for HDFS and YARN, we support falling back to native permission check if we don’t have corresponding permission in Ranger. So in your case, since there were no permissions in Ranger for “asma” to the folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t have native posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed creating the folder and the audit would should that hadoop-acl allowed to create the folder.
This also answers yours previous question why we want to make umask=077 and chmod –r 000 to all application folders to be managed by Ranger. So if there are no Ranger policies, then we want to hadoop also to deny.
With the recent deny feature, you can explicitly “deny” “asma” or any group from creating/writing. Or you could deny all, but exclude “developer’ and “sadaf” from the deny users.
In the future release, I feel, we should provide a way to mark certain folders to be managed exclusively by Ranger. And that will remove a lot of confusion and also make the policy management more predictable.
Does it answer your question?
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 8:59 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco!
I created a directory /mjd with following permissions
drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd
Then i made a policy with following permissions
Datascientist group has one user asma and developer group has one user named haniya and sadaf has no group.
So when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1
mkdir: Permission denied: user=asma, access=WRITE, inode="/mjd/a1":hduser:supergroup:drwxr-xr-x
And audit of this command is as follow
Service
Policy IDEvent TimeUserName / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
--12/02/2015 09:46:23 AMasmahdfsRepo
/mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Hi Bosco*,*
I have tested Deny, Allow , Exclude from deny and exclude from Allow
permissions. Following are details of my findings.
*Test Cases*
*Developer Group: *Roger, Smith
*Data Scientist Group: *Mike, Clark
Hadoop Resource : */perm*
Ranger Policy
[image: Inline image 2]
Developers can do nothing having no read, write and execute permissions
except Roger who has full permissions .All users of group Data-Scientist
have permissions read, write and execute permissions except *mike* who
can't write.
Test Cases
Test_ID
User
Group
Command
Expected
Actual
1
Roger
Developers
Hdfs dfs -ls /perm
Allowed
Allowed
2
Roger
Developers
Hdfs dfs -mkdir /perm/r
Allowed
Allowed
3
Smith
Developers
Hdfs dfs -mkdir /perm/s
Denied
Denied
4
Smith
Developers
Hdfs dfs -ls /perm
Denied
Denied
5
Clark
Data-Scientis
Hdfs dfs -ls /perm
Allowed
Allowed
6
Clark
Data-Scientis
Hdfs dfs -mkdir /perm/c
Allowed
Allowed
7
Mike
Data-Scientis
Hdfs dfs -mkdir /perm/m
Alowed
Allowed
8
Mike
Data-Scientis
Hdfs dfs -ls /perm
Allowed
Allowed
On Thu, Dec 3, 2015 at 11:24 PM, Hafiz Mujadid <ha...@gmail.com>
wrote:
> Hi Bosco,
>
> Thanks for your response, I am testing new feature of ranger Deny,Allow.
> will send you my findings in short.
>
> Thanks
>
> On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
>
>> >I want to know why audits are showing that it is because of hadoop-acl
>> not ranger-acl?
>> Hafiz, this is a good question and we should probably document it or come
>> with a blog for this.
>>
>> Only for HDFS and YARN, we support falling back to native permission
>> check if we don’t have corresponding permission in Ranger. So in your case,
>> since there were no permissions in Ranger for “asma” to the folder “/mjd”,
>> we went and checked hadoop-acl. And since even hadoop didn’t have native
>> posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the
>> last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
>> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
>> creating the folder and the audit would should that hadoop-acl allowed to
>> create the folder.
>>
>> This also answers yours previous question why we want to make umask=077
>> and chmod –r 000 to all application folders to be managed by Ranger. So if
>> there are no Ranger policies, then we want to hadoop also to deny.
>>
>> With the recent deny feature, you can explicitly “deny” “asma” or any
>> group from creating/writing. Or you could deny all, but exclude “developer’
>> and “sadaf” from the deny users.
>>
>> In the future release, I feel, we should provide a way to mark certain
>> folders to be managed exclusively by Ranger. And that will remove a lot of
>> confusion and also make the policy management more predictable.
>>
>> Does it answer your question?
>>
>> Bosco
>>
>>
>> From: Hafiz Mujadid <ha...@gmail.com>
>> Reply-To: <us...@ranger.incubator.apache.org>
>> Date: Tuesday, December 1, 2015 at 8:59 PM
>>
>> To: <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Hi Bosco!
>>
>> I created a directory /mjd with following permissions
>> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>>
>> Then i made a policy with following permissions
>> [image: Inline image 1]
>> Datascientist group has one user asma and developer group has one user
>> named haniya and sadaf has no group.
>>
>> So when i run following command
>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
>> *mkdir: Permission denied: user=asma, access=WRITE,
>> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>>
>>
>>
>> *And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
>> / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count--12/02/2015
>> 09:46:23 AMasma
>> hdfsRepo
>> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
>> I want to know why audits are showing that it is because of hadoop-acl
>> not ranger-acl?
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>>> You don’t need to. Since auditing is working, you can check who gave the
>>> permission without 000
>>>
>>> We recommend giving 000 at HDFS level, because Ranger by default falls
>>> back to HDFS permission. So for all folders you want to Ranger to be
>>> exclusive, you give as minimal permission as possible.
>>>
>>> I think, we should also make it configurable in Ranger. Where you can
>>> tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t
>>> have to worry about HDFS level ACLs.
>>>
>>> The reason you don’t want Ranger to manage everything because there are
>>> folders like tmp and user folders which want the system and user to manage
>>> themselves. But for application folders like Hive warehouse, you should let
>>> Ranger manage it.
>>>
>>> Bosco
>>>
>>> From: Hafiz Mujadid <ha...@gmail.com>
>>> Reply-To: <us...@ranger.incubator.apache.org>
>>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>>
>>> To: <us...@ranger.incubator.apache.org>
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Hi,
>>>
>>> Bosco, I noticed group level permission works when we set hadoop
>>> permissions to 000. I am just curious why it is so ?
>>>
>>> is it always necessary to set hadoop permissions to 000 for ranger to
>>> work?
>>>
>>> thanks
>>>
>>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <
>>> hafizmujadid00@gmail.com> wrote:
>>>
>>>> Bosco, I have tried both mysql db and solr as well, only plugin related
>>>> auditing is being shown
>>>>
>>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> Yes, you should fix audit first. That will help in debugging these
>>>>> issues also.
>>>>>
>>>>> BTW, are you using Solr or DB?
>>>>>
>>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new
>>>>> package for setting up Solr. It is available as attachment in
>>>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions
>>>>> are in
>>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>>
>>>>> Give it a try.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>>
>>>>> To: "user@ranger.incubator.apache.org" <
>>>>> user@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Hafiz,
>>>>>
>>>>> Few things to check:
>>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>>> 2. Can you disable this policy and try mkdir?
>>>>>
>>>>> Fixing the issue with audit will help; audit log will have the details
>>>>> of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>>> ranger-acl, the policy-ID that determined the access).
>>>>>
>>>>> Madhan
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>>> user@ranger.incubator.apache.org>
>>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>>> To: "user@ranger.incubator.apache.org" <
>>>>> user@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Bosco,
>>>>>
>>>>> I have followed above steps
>>>>>
>>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>> 2. changed the umask so newly created folder or files have
>>>>> following permissions
>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>> 3. i changed the ownership of all folders in hdfs with
>>>>> hduser:hadoop
>>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>>
>>>>>
>>>>> but still group level permissions are not working.
>>>>>
>>>>> my audits are not working, i am trying to figure out the issue with
>>>>> audits. i will let you know when audits are available.
>>>>>
>>>>>
>>>>> thanks
>>>>>
>>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>>> hafizmujadid00@gmail.com> wrote:
>>>>>
>>>>>> Bosco,
>>>>>>
>>>>>> I have followed above steps
>>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>>> changed the umask so newly created folder or files have following
>>>>>> permissions
>>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>>
>>>>>> but still group level permissions are not working.
>>>>>>
>>>>>>
>>>>>> my audits are not working, i am trying to figure out the issue with
>>>>>> audits. i will let you know when audits are available.
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Can you check Ranger Audits?
>>>>>>>
>>>>>>> Also, do couple of things:
>>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>>
>>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>>
>>>>>>> Bosco
>>>>>>>
>>>>>>>
>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Yes Bosco, directory is being created.
>>>>>>>
>>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Bosco
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>>
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> I am trying to apply permission on an ldap group but it's not
>>>>>>>> working
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>>
>>>>>>>> But when i run following command
>>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>>
>>>>>>>> i works successfully
>>>>>>>> what is the issue? ldap users and groups are synced correctly as
>>>>>>>> when i run the command *hdfs groups asma* it returns correct
>>>>>>>> group
>>>>>>>> asma : datascientist
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards: HAFIZ MUJADID
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>>
>
>
> --
> Regards: HAFIZ MUJADID
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Hi Bosco,
Thanks for your response, I am testing new feature of ranger Deny,Allow.
will send you my findings in short.
Thanks
On Thu, Dec 3, 2015 at 10:40 PM, Don Bosco Durai <bo...@apache.org> wrote:
> >I want to know why audits are showing that it is because of hadoop-acl
> not ranger-acl?
> Hafiz, this is a good question and we should probably document it or come
> with a blog for this.
>
> Only for HDFS and YARN, we support falling back to native permission check
> if we don’t have corresponding permission in Ranger. So in your case, since
> there were no permissions in Ranger for “asma” to the folder “/mjd”, we
> went and checked hadoop-acl. And since even hadoop didn’t have native posix
> ACL for asma for the folder /mjd, it denied it. Since hadoop was the last
> one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS
> level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed
> creating the folder and the audit would should that hadoop-acl allowed to
> create the folder.
>
> This also answers yours previous question why we want to make umask=077
> and chmod –r 000 to all application folders to be managed by Ranger. So if
> there are no Ranger policies, then we want to hadoop also to deny.
>
> With the recent deny feature, you can explicitly “deny” “asma” or any
> group from creating/writing. Or you could deny all, but exclude “developer’
> and “sadaf” from the deny users.
>
> In the future release, I feel, we should provide a way to mark certain
> folders to be managed exclusively by Ranger. And that will remove a lot of
> confusion and also make the policy management more predictable.
>
> Does it answer your question?
>
> Bosco
>
>
> From: Hafiz Mujadid <ha...@gmail.com>
> Reply-To: <us...@ranger.incubator.apache.org>
> Date: Tuesday, December 1, 2015 at 8:59 PM
>
> To: <us...@ranger.incubator.apache.org>
> Subject: Re: Group level permission are not working in ranger
>
> Hi Bosco!
>
> I created a directory /mjd with following permissions
> *drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
>
> Then i made a policy with following permissions
> [image: Inline image 1]
> Datascientist group has one user asma and developer group has one user
> named haniya and sadaf has no group.
>
> So when i run following command
> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
> *mkdir: Permission denied: user=asma, access=WRITE,
> inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
>
>
>
> *And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
> / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count--12/02/2015
> 09:46:23 AMasma
> hdfsRepo
> /mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
> I want to know why audits are showing that it is because of hadoop-acl not
> ranger-acl?
>
> Thanks
>
>
>
>
>
> On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
>
>> You don’t need to. Since auditing is working, you can check who gave the
>> permission without 000
>>
>> We recommend giving 000 at HDFS level, because Ranger by default falls
>> back to HDFS permission. So for all folders you want to Ranger to be
>> exclusive, you give as minimal permission as possible.
>>
>> I think, we should also make it configurable in Ranger. Where you can
>> tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t
>> have to worry about HDFS level ACLs.
>>
>> The reason you don’t want Ranger to manage everything because there are
>> folders like tmp and user folders which want the system and user to manage
>> themselves. But for application folders like Hive warehouse, you should let
>> Ranger manage it.
>>
>> Bosco
>>
>> From: Hafiz Mujadid <ha...@gmail.com>
>> Reply-To: <us...@ranger.incubator.apache.org>
>> Date: Tuesday, December 1, 2015 at 1:31 PM
>>
>> To: <us...@ranger.incubator.apache.org>
>> Subject: Re: Group level permission are not working in ranger
>>
>> Hi,
>>
>> Bosco, I noticed group level permission works when we set hadoop
>> permissions to 000. I am just curious why it is so ?
>>
>> is it always necessary to set hadoop permissions to 000 for ranger to
>> work?
>>
>> thanks
>>
>> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <hafizmujadid00@gmail.com
>> > wrote:
>>
>>> Bosco, I have tried both mysql db and solr as well, only plugin related
>>> auditing is being shown
>>>
>>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>>
>>>> Yes, you should fix audit first. That will help in debugging these
>>>> issues also.
>>>>
>>>> BTW, are you using Solr or DB?
>>>>
>>>> Recommendation is to use Solr. Yesterday, I have uploaded a new package
>>>> for setting up Solr. It is available as attachment in
>>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions are
>>>> in
>>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>>
>>>> Give it a try.
>>>>
>>>> Thanks
>>>>
>>>> Bosco
>>>>
>>>>
>>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>>
>>>> To: "user@ranger.incubator.apache.org" <
>>>> user@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Hafiz,
>>>>
>>>> Few things to check:
>>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>>> 2. Can you disable this policy and try mkdir?
>>>>
>>>> Fixing the issue with audit will help; audit log will have the details
>>>> of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>>> ranger-acl, the policy-ID that determined the access).
>>>>
>>>> Madhan
>>>>
>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>> Reply-To: "user@ranger.incubator.apache.org" <
>>>> user@ranger.incubator.apache.org>
>>>> Date: Monday, November 30, 2015 at 6:16 AM
>>>> To: "user@ranger.incubator.apache.org" <
>>>> user@ranger.incubator.apache.org>
>>>> Subject: Re: Group level permission are not working in ranger
>>>>
>>>> Bosco,
>>>>
>>>> I have followed above steps
>>>>
>>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>> 2. changed the umask so newly created folder or files have
>>>> following permissions
>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>> 3. i changed the ownership of all folders in hdfs with hduser:hadoop
>>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>>
>>>>
>>>> but still group level permissions are not working.
>>>>
>>>> my audits are not working, i am trying to figure out the issue with
>>>> audits. i will let you know when audits are available.
>>>>
>>>>
>>>> thanks
>>>>
>>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <
>>>> hafizmujadid00@gmail.com> wrote:
>>>>
>>>>> Bosco,
>>>>>
>>>>> I have followed above steps
>>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>>> changed the umask so newly created folder or files have following
>>>>> permissions
>>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>>
>>>>> but still group level permissions are not working.
>>>>>
>>>>>
>>>>> my audits are not working, i am trying to figure out the issue with
>>>>> audits. i will let you know when audits are available.
>>>>>
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Can you check Ranger Audits?
>>>>>>
>>>>>> Also, do couple of things:
>>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>>
>>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>>
>>>>>> Yes Bosco, directory is being created.
>>>>>>
>>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> What is happening here? Is the directory getting created?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Bosco
>>>>>>>
>>>>>>>
>>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>>> Subject: Group level permission are not working in ranger
>>>>>>>
>>>>>>> Hi all
>>>>>>>
>>>>>>> I am trying to apply permission on an ldap group but it's not working
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>> But when i run following command
>>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>>
>>>>>>> i works successfully
>>>>>>> what is the issue? ldap users and groups are synced correctly as
>>>>>>> when i run the command *hdfs groups asma* it returns correct
>>>>>>> group
>>>>>>> asma : datascientist
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards: HAFIZ MUJADID
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>>
>
>
> --
> Regards: HAFIZ MUJADID
>
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Don Bosco Durai <bo...@apache.org>.
>I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Hafiz, this is a good question and we should probably document it or come with a blog for this.
Only for HDFS and YARN, we support falling back to native permission check if we don’t have corresponding permission in Ranger. So in your case, since there were no permissions in Ranger for “asma” to the folder “/mjd”, we went and checked hadoop-acl. And since even hadoop didn’t have native posix ACL for asma for the folder /mjd, it denied it. Since hadoop was the last one to deny, you saw “hadoop-acl” in the audit record. If in the HDFS level, you had given rwx-rwx-rwx ACLs, then HDFS would have allowed creating the folder and the audit would should that hadoop-acl allowed to create the folder.
This also answers yours previous question why we want to make umask=077 and chmod –r 000 to all application folders to be managed by Ranger. So if there are no Ranger policies, then we want to hadoop also to deny.
With the recent deny feature, you can explicitly “deny” “asma” or any group from creating/writing. Or you could deny all, but exclude “developer’ and “sadaf” from the deny users.
In the future release, I feel, we should provide a way to mark certain folders to be managed exclusively by Ranger. And that will remove a lot of confusion and also make the policy management more predictable.
Does it answer your question?
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 8:59 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi Bosco!
I created a directory /mjd with following permissions
drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd
Then i made a policy with following permissions
Datascientist group has one user asma and developer group has one user named haniya and sadaf has no group.
So when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1
mkdir: Permission denied: user=asma, access=WRITE, inode="/mjd/a1":hduser:supergroup:drwxr-xr-x
And audit of this command is as follow
Service
Policy IDEvent TimeUserName / TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent Count
--12/02/2015 09:46:23 AMasmahdfsRepo
/mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
I want to know why audits are showing that it is because of hadoop-acl not ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Hafiz Mujadid <ha...@gmail.com>.
Hi Bosco!
I created a directory /mjd with following permissions
*drwxr-xr-x - hduser supergroup 0 2015-12-02 09:44 /mjd*
Then i made a policy with following permissions
[image: Inline image 1]
Datascientist group has one user asma and developer group has one user
named haniya and sadaf has no group.
So when i run following command
*HADOOP_USER_NAME=asma hdfs dfs -mkdir /mjd/a1*
*mkdir: Permission denied: user=asma, access=WRITE,
inode="/mjd/a1":hduser:supergroup:drwxr-xr-x*
*And audit of this command is as follow*ServicePolicy IDEvent TimeUserName
/ TypeResource NameAccess TypeResultAccess EnforcerClient IPEvent
Count--12/02/2015
09:46:23 AMasma
hdfsRepo
/mjd/a1WRITEDeniedhadoop-acl192.168.23.1051
I want to know why audits are showing that it is because of hadoop-acl not
ranger-acl?
Thanks
On Wed, Dec 2, 2015 at 9:37 AM, Don Bosco Durai <bo...@apache.org> wrote:
> You don’t need to. Since auditing is working, you can check who gave the
> permission without 000
>
> We recommend giving 000 at HDFS level, because Ranger by default falls
> back to HDFS permission. So for all folders you want to Ranger to be
> exclusive, you give as minimal permission as possible.
>
> I think, we should also make it configurable in Ranger. Where you can tell
> Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have
> to worry about HDFS level ACLs.
>
> The reason you don’t want Ranger to manage everything because there are
> folders like tmp and user folders which want the system and user to manage
> themselves. But for application folders like Hive warehouse, you should let
> Ranger manage it.
>
> Bosco
>
> From: Hafiz Mujadid <ha...@gmail.com>
> Reply-To: <us...@ranger.incubator.apache.org>
> Date: Tuesday, December 1, 2015 at 1:31 PM
>
> To: <us...@ranger.incubator.apache.org>
> Subject: Re: Group level permission are not working in ranger
>
> Hi,
>
> Bosco, I noticed group level permission works when we set hadoop
> permissions to 000. I am just curious why it is so ?
>
> is it always necessary to set hadoop permissions to 000 for ranger to
> work?
>
> thanks
>
> On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com>
> wrote:
>
>> Bosco, I have tried both mysql db and solr as well, only plugin related
>> auditing is being shown
>>
>> On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org>
>> wrote:
>>
>>> Yes, you should fix audit first. That will help in debugging these
>>> issues also.
>>>
>>> BTW, are you using Solr or DB?
>>>
>>> Recommendation is to use Solr. Yesterday, I have uploaded a new package
>>> for setting up Solr. It is available as attachment in
>>> https://issues.apache.org/jira/browse/RANGER-728. The instructions are
>>> in
>>> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
>>>
>>> Give it a try.
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
>>> From: Madhan Neethiraj <mn...@hortonworks.com>
>>> Reply-To: <us...@ranger.incubator.apache.org>
>>> Date: Monday, November 30, 2015 at 8:57 AM
>>>
>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org
>>> >
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Hafiz,
>>>
>>> Few things to check:
>>> 1. Do you have another policy in Ranger that allows WRITE access?
>>> 2. Can you disable this policy and try mkdir?
>>>
>>> Fixing the issue with audit will help; audit log will have the details
>>> of how the access was allowed (hadoop-acl or ranger-acl; in case of
>>> ranger-acl, the policy-ID that determined the access).
>>>
>>> Madhan
>>>
>>> From: Hafiz Mujadid <ha...@gmail.com>
>>> Reply-To: "user@ranger.incubator.apache.org" <
>>> user@ranger.incubator.apache.org>
>>> Date: Monday, November 30, 2015 at 6:16 AM
>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org
>>> >
>>> Subject: Re: Group level permission are not working in ranger
>>>
>>> Bosco,
>>>
>>> I have followed above steps
>>>
>>> 1. drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>> 2. changed the umask so newly created folder or files have following
>>> permissions
>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>> 3. i changed the ownership of all folders in hdfs with hduser:hadoop
>>> 4. ran the command hdfs dfs -chmod -R 000 /pg
>>>
>>>
>>> but still group level permissions are not working.
>>>
>>> my audits are not working, i am trying to figure out the issue with
>>> audits. i will let you know when audits are available.
>>>
>>>
>>> thanks
>>>
>>> On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <hafizmujadid00@gmail.com
>>> > wrote:
>>>
>>>> Bosco,
>>>>
>>>> I have followed above steps
>>>> drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
>>>> changed the umask so newly created folder or files have following
>>>> permissions
>>>> d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
>>>> i changed the ownership of all folders in hdfs with hduser:hadoop
>>>>
>>>> but still group level permissions are not working.
>>>>
>>>>
>>>> my audits are not working, i am trying to figure out the issue with
>>>> audits. i will let you know when audits are available.
>>>>
>>>>
>>>> thanks
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>>
>>>>> Can you check Ranger Audits?
>>>>>
>>>>> Also, do couple of things:
>>>>> 1. hdfs dfs -ls /pg (check the HDFS level permissions)
>>>>> 2. In HDFS settngs, set the umask to 700 and restart name node.
>>>>> 3. hdfs dfs -chown hdfs:hdfs /pg
>>>>> 4. hdfs dfs -chmod -R 000 /pg
>>>>>
>>>>> For all user folders, e.g. /app/hive, do #3 and #4 as above.
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>> Date: Sunday, November 29, 2015 at 8:29 PM
>>>>> To: <us...@ranger.incubator.apache.org>
>>>>> Subject: Re: Group level permission are not working in ranger
>>>>>
>>>>> Yes Bosco, directory is being created.
>>>>>
>>>>> On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> What is happening here? Is the directory getting created?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Bosco
>>>>>>
>>>>>>
>>>>>> From: Hafiz Mujadid <ha...@gmail.com>
>>>>>> Reply-To: <us...@ranger.incubator.apache.org>
>>>>>> Date: Sunday, November 29, 2015 at 1:44 PM
>>>>>> To: <us...@ranger.incubator.apache.org>
>>>>>> Subject: Group level permission are not working in ranger
>>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> I am trying to apply permission on an ldap group but it's not working
>>>>>>
>>>>>> [image: Inline image 1]
>>>>>>
>>>>>>
>>>>>> But when i run following command
>>>>>> *HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b*
>>>>>>
>>>>>> i works successfully
>>>>>> what is the issue? ldap users and groups are synced correctly as when
>>>>>> i run the command *hdfs groups asma* it returns correct group
>>>>>> asma : datascientist
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards: HAFIZ MUJADID
>>>>
>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
>
--
Regards: HAFIZ MUJADID
Re: Group level permission are not working in ranger
Posted by Don Bosco Durai <bo...@apache.org>.
You don’t need to. Since auditing is working, you can check who gave the permission without 000
We recommend giving 000 at HDFS level, because Ranger by default falls back to HDFS permission. So for all folders you want to Ranger to be exclusive, you give as minimal permission as possible.
I think, we should also make it configurable in Ranger. Where you can tell Ranger for these folders, it shouldn’t fall back to HDFS. So you don’t have to worry about HDFS level ACLs.
The reason you don’t want Ranger to manage everything because there are folders like tmp and user folders which want the system and user to manage themselves. But for application folders like Hive warehouse, you should let Ranger manage it.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, December 1, 2015 at 1:31 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hi,
Bosco, I noticed group level permission works when we set hadoop permissions to 000. I am just curious why it is so ?
is it always necessary to set hadoop permissions to 000 for ranger to work?
thanks
On Mon, Nov 30, 2015 at 10:59 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco, I have tried both mysql db and solr as well, only plugin related auditing is being shown
On Mon, Nov 30, 2015 at 10:53 PM, Don Bosco Durai <bo...@apache.org> wrote:
Yes, you should fix audit first. That will help in debugging these issues also.
BTW, are you using Solr or DB?
Recommendation is to use Solr. Yesterday, I have uploaded a new package for setting up Solr. It is available as attachment in https://issues.apache.org/jira/browse/RANGER-728. The instructions are in https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
Give it a try.
Thanks
Bosco
From: Madhan Neethiraj <mn...@hortonworks.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 8:57 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Hafiz,
Few things to check:
1. Do you have another policy in Ranger that allows WRITE access?
2. Can you disable this policy and try mkdir?
Fixing the issue with audit will help; audit log will have the details of how the access was allowed (hadoop-acl or ranger-acl; in case of ranger-acl, the policy-ID that determined the access).
Madhan
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Date: Monday, November 30, 2015 at 6:16 AM
To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
ran the command hdfs dfs -chmod -R 000 /pg
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 7:13 PM, Hafiz Mujadid <ha...@gmail.com> wrote:
Bosco,
I have followed above steps
drwxr-xr-x - hduser hadoop 0 2015-11-30 18:49 /pg
changed the umask so newly created folder or files have following permissions
d---rwxrwx - asma hadoop 0 2015-11-30 19:03 /pg/b
i changed the ownership of all folders in hdfs with hduser:hadoop
but still group level permissions are not working.
my audits are not working, i am trying to figure out the issue with audits. i will let you know when audits are available.
thanks
On Mon, Nov 30, 2015 at 9:34 AM, Don Bosco Durai <bo...@apache.org> wrote:
Can you check Ranger Audits?
Also, do couple of things:
1. hdfs dfs -ls /pg (check the HDFS level permissions)
2. In HDFS settngs, set the umask to 700 and restart name node.
3. hdfs dfs -chown hdfs:hdfs /pg
4. hdfs dfs -chmod -R 000 /pg
For all user folders, e.g. /app/hive, do #3 and #4 as above.
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 8:29 PM
To: <us...@ranger.incubator.apache.org>
Subject: Re: Group level permission are not working in ranger
Yes Bosco, directory is being created.
On Mon, Nov 30, 2015 at 2:47 AM, Don Bosco Durai <bo...@apache.org> wrote:
What is happening here? Is the directory getting created?
Thanks
Bosco
From: Hafiz Mujadid <ha...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Sunday, November 29, 2015 at 1:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Group level permission are not working in ranger
Hi all
I am trying to apply permission on an ldap group but it's not working
But when i run following command
HADOOP_USER_NAME=asma hdfs dfs -mkdir /pg/b
i works successfully
what is the issue? ldap users and groups are synced correctly as when i run the command hdfs groups asma it returns correct group
asma : datascientist
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID
--
Regards: HAFIZ MUJADID