You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Loïc Chanel <lo...@telecomnancy.net> on 2016/09/06 08:31:16 UTC
User running job in forbidden queue
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
quite a big issue.
Using Ranger to handle which user can submit job to which queue I
authorized user "test" to submit jobs on queue "test" only - with the
property ranger.add-yarn-authorization set to false in ranger-yarn-security.
But even with these settings when user "test" submit a job it goes in the
"default" queue - to which he shouldn't be able to submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Re: Antwort: Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Markus,
In addition to Bosco's information, I highly recommend you the following
link :
https://github.com/abajwa-hw/security-workshops/blob/master/Setup-ranger-23.md
It is quite helpful to have a precise idea of how to secure a Hadoop
cluster and the author seems to keep it up to date -- it evolved a lot
since last year when I was working on the topic. Hope this will help you as
much as it helps me.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-07 1:39 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
> Hi Markus
>
>
>
> Multi-tenancy can be tricky based on use cases. In Ranger, at the admin
> level you can setup multi-tenancy using “Delegated Admin” feature. The
> Ranger 0.5 user guides talks about it briefly https://cwiki.apache.org/
> confluence/display/RANGER/Apache+Ranger+0.5+-+User+Guide and also there
> is more explanation in this forum discussion
> https://community.hortonworks.com/questions/17782/about-
> delegate-admin-in-ranger.html.
>
>
>
> In Ranger 0.6 we have introduced Row level filtering, which can further
> restrict the users for the rows they have access to it. Madhan from the
> Ranger team has written a nice document on both row level filtering and
> masking, which work for both Hive and Spark SQL (when used with LLAP)
> https://cwiki.apache.org/confluence/display/RANGER/Row-
> level+filtering+and+column-masking+using+Apache+Ranger+
> policies+in+Apache+Hive
>
>
>
> Only Audit doesn’t have filtering, however it is role based access
> controlled, so only certain users can access audit logs.
>
>
>
> So it comes back to use cases and deployment scenarios. If you can cleanly
> separate tenants by HDFS folders, Hive & HBase databases then you can
> delegate top level “delegated admin” privileges to tenant owners so that
> they can manage further permissions within their group. Since they can’t
> give permissions to resources they don’t have permission, you are pretty
> safe.
>
>
>
> If your data is inter-mingled or if you need more dynamic access control,
> then row-level filtering will be the best case, but you have to ensure
> there is a tenant id column or can be derived with join to do the filtering.
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
>
>
> *From: *<Ma...@fiduciagad.de>
> *Reply-To: *<us...@ranger.incubator.apache.org>
> *Date: *Tuesday, September 6, 2016 at 10:23 AM
> *To: *<us...@ranger.incubator.apache.org>
> *Subject: *Antwort: Re: User running job in forbidden queue
>
>
>
> Hi Bosco,
>
> is there a guide or a best practice document for multi-tenancy ? Can I
> really separate different use cases on the same cluster without any
> security doubts?
>
> Best regards,
>
> Markus
>
> Fiducia & GAD IT AG | www.fiduciagad.de
> AG Frankfurt a. M. HRB 102381 | Sitz der Gesellschaft: Hahnstr. 48, 60528
> Frankfurt a. M. | USt-IdNr. DE 143582320
> Vorstand: Klaus-Peter Bruns (Vorsitzender), Claus-Dieter Toben (stv.
> Vorsitzender),
> Jens-Olaf Bartels, Martin Beyer, Jörg Dreinhöfer, Wolfgang Eckert, Carsten
> Pfläging, Jörg Staff
> Vorsitzender des Aufsichtsrats: Jürgen Brinkmann
>
> [image: naktiv: Details verbergen für Don Bosco Durai ---06.09.2016
> 18:41:06---Hi]Don Bosco Durai ---06.09.2016 18:41:06---Hi Loïc
>
>
> Von: Don Bosco Durai <bo...@apache.org>
> An: <us...@ranger.incubator.apache.org>
> Datum: 06.09.2016 18:41
> Betreff: Re: User running job in forbidden queue
>
> ------------------------------
>
>
>
>
> Hi Loïc
>
> Just curious, was the audit log helpful? I understand, this could be
> frustrating, so during the last couple of releases, in the audit logs, we
> have added more information to help admins understand which policy gave the
> permission to access (or deny).
>
> However, in your case, since it was denied, there might have been no
> policy, but the resource field should have given the resource name as
> “root.test”. If not, we should look into this.
>
> Any suggestions to improve is welcomed…
>
> Thanks
>
> Bosco
>
>
>
>
> *From: *Loïc Chanel <lo...@telecomnancy.net>
> *Reply-To: *<us...@ranger.incubator.apache.org>
> *Date: *Tuesday, September 6, 2016 at 8:51 AM
> *To: *<us...@ranger.incubator.apache.org>
> *Subject: *Re: User running job in forbidden queue
>
> And now I feel like a complete idiot because my actual problem was the
> fact that in Ranger policies I wrote "test" instead of "root.test".
> Sorry for the spam, then.
>
> Regards,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
> 2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Actually, I ran some further tests and the property
> ranger.add-yarn-authorization set to false in ranger-yarn-security seems to
> prevent anyone to run jobs in any queue as my user "test" cannot submit a
> job into "test" queue according to YARN.
> Anyone encountered the same issue ?
>
> FYI, I am using an HDP 2.4 stack.
>
> Regards,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
> 2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Hi all,
>
> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
> quite a big issue.
> Using Ranger to handle which user can submit job to which queue I
> authorized user "test" to submit jobs on queue "test" only - with the
> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
> But even with these settings when user "test" submit a job it goes in the
> "default" queue - to which he shouldn't be able to submit jobs.
>
> Do you see what I miss here ?
> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>
> Thanks in advance for your inputs,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
>
>
Re: Antwort: Re: User running job in forbidden queue
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Markus
Multi-tenancy can be tricky based on use cases. In Ranger, at the admin level you can setup multi-tenancy using “Delegated Admin” feature. The Ranger 0.5 user guides talks about it briefly https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5+-+User+Guide and also there is more explanation in this forum discussion https://community.hortonworks.com/questions/17782/about-delegate-admin-in-ranger.html.
In Ranger 0.6 we have introduced Row level filtering, which can further restrict the users for the rows they have access to it. Madhan from the Ranger team has written a nice document on both row level filtering and masking, which work for both Hive and Spark SQL (when used with LLAP) https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive
Only Audit doesn’t have filtering, however it is role based access controlled, so only certain users can access audit logs.
So it comes back to use cases and deployment scenarios. If you can cleanly separate tenants by HDFS folders, Hive & HBase databases then you can delegate top level “delegated admin” privileges to tenant owners so that they can manage further permissions within their group. Since they can’t give permissions to resources they don’t have permission, you are pretty safe.
If your data is inter-mingled or if you need more dynamic access control, then row-level filtering will be the best case, but you have to ensure there is a tenant id column or can be derived with join to do the filtering.
Thanks
Bosco
From: <Ma...@fiduciagad.de>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, September 6, 2016 at 10:23 AM
To: <us...@ranger.incubator.apache.org>
Subject: Antwort: Re: User running job in forbidden queue
Hi Bosco,
is there a guide or a best practice document for multi-tenancy ? Can I really separate different use cases on the same cluster without any security doubts?
Best regards,
Markus
Fiducia & GAD IT AG | www.fiduciagad.de
AG Frankfurt a. M. HRB 102381 | Sitz der Gesellschaft: Hahnstr. 48, 60528 Frankfurt a. M. | USt-IdNr. DE 143582320
Vorstand: Klaus-Peter Bruns (Vorsitzender), Claus-Dieter Toben (stv. Vorsitzender),
Jens-Olaf Bartels, Martin Beyer, Jörg Dreinhöfer, Wolfgang Eckert, Carsten Pfläging, Jörg Staff
Vorsitzender des Aufsichtsrats: Jürgen Brinkmann
Don Bosco Durai ---06.09.2016 18:41:06---Hi Loïc
Von: Don Bosco Durai <bo...@apache.org>
An: <us...@ranger.incubator.apache.org>
Datum: 06.09.2016 18:41
Betreff: Re: User running job in forbidden queue
Hi Loïc
Just curious, was the audit log helpful? I understand, this could be frustrating, so during the last couple of releases, in the audit logs, we have added more information to help admins understand which policy gave the permission to access (or deny).
However, in your case, since it was denied, there might have been no policy, but the resource field should have given the resource name as “root.test”. If not, we should look into this.
Any suggestions to improve is welcomed…
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, September 6, 2016 at 8:51 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: User running job in forbidden queue
And now I feel like a complete idiot because my actual problem was the fact that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Actually, I ran some further tests and the property ranger.add-yarn-authorization set to false in ranger-yarn-security seems to prevent anyone to run jobs in any queue as my user "test" cannot submit a job into "test" queue according to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I quite a big issue.
Using Ranger to handle which user can submit job to which queue I authorized user "test" to submit jobs on queue "test" only - with the property ranger.add-yarn-authorization set to false in ranger-yarn-security.
But even with these settings when user "test" submit a job it goes in the "default" queue - to which he shouldn't be able to submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Antwort: Re: User running job in forbidden queue
Posted by Ma...@fiduciagad.de.
Hi Bosco,
is there a guide or a best practice document for multi-tenancy ? Can I
really separate different use cases on the same cluster without any
security doubts?
Best regards,
Markus
Fiducia & GAD IT AG | www.fiduciagad.de
AG Frankfurt a. M. HRB 102381 | Sitz der Gesellschaft: Hahnstr. 48, 60528
Frankfurt a. M. | USt-IdNr. DE 143582320
Vorstand: Klaus-Peter Bruns (Vorsitzender), Claus-Dieter Toben (stv.
Vorsitzender),
Jens-Olaf Bartels, Martin Beyer, Jörg Dreinhöfer, Wolfgang Eckert, Carsten
Pfläging, Jörg Staff
Vorsitzender des Aufsichtsrats: Jürgen Brinkmann
Von: Don Bosco Durai <bo...@apache.org>
An: <us...@ranger.incubator.apache.org>
Datum: 06.09.2016 18:41
Betreff: Re: User running job in forbidden queue
Hi Loïc
Just curious, was the audit log helpful? I understand, this could be
frustrating, so during the last couple of releases, in the audit logs, we
have added more information to help admins understand which policy gave the
permission to access (or deny).
However, in your case, since it was denied, there might have been no
policy, but the resource field should have given the resource name as
“root.test”. If not, we should look into this.
Any suggestions to improve is welcomed…
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, September 6, 2016 at 8:51 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: User running job in forbidden queue
And now I feel like a complete idiot because my actual problem was the fact
that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Actually, I ran some further tests and the property
ranger.add-yarn-authorization set to false in ranger-yarn-security
seems to prevent anyone to run jobs in any queue as my user "test"
cannot submit a job into "test" queue according to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
>:
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some
tests I quite a big issue.
Using Ranger to handle which user can submit job to which queue
I authorized user "test" to submit jobs on queue "test" only -
with the property ranger.add-yarn-authorization set to false in
ranger-yarn-security.
But even with these settings when user "test" submit a job it
goes in the "default" queue - to which he shouldn't be able to
submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug
logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
That's exactly what I was thinking, that's why I couldn't see any
improvement of the feature but only a slight upgrade of the user's brain ;-)
Still, the validation option sounds like a good idea.
Regards,
Loïcc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-07 19:32 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
> Hi Loïc
>
>
>
> > A possible improvement would be to prevent a user from creating a policy
> for a queue that doesn't exist.
>
> > Tell me if it sounds feasible, and I'll create the corresponding Jira.
>
>
>
> I am not sure this will work. One of Ranger flexible feature is to be able
> to create policies on resources which are not yet created. We could give
> validation options to verify if the resource is present or not. This could
> help in debugging.
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
> *From: *Loïc Chanel <lo...@telecomnancy.net>
> *Reply-To: *<us...@ranger.incubator.apache.org>
> *Date: *Wednesday, September 7, 2016 at 1:07 AM
>
> *To: *<us...@ranger.incubator.apache.org>
> *Subject: *Re: User running job in forbidden queue
>
>
>
> Bosco,
>
>
>
> I think the audit log would have been a great indicator of my mistake, but
> another problem I have to work on is the fact that I haven't any audit logs
> but the ones in the Plugins tab ;-)
>
> This is what happen when you do not build the cluster yourself ;-)
>
>
>
> A possible improvement would be to prevent a user from creating a policy
> for a queue that doesn't exist.
>
> Tell me if it sounds feasible, and I'll create the corresponding Jira.
>
>
>
> Regards,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
> 2016-09-06 18:40 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
>
> Hi Loïc
>
>
>
> Just curious, was the audit log helpful? I understand, this could be
> frustrating, so during the last couple of releases, in the audit logs, we
> have added more information to help admins understand which policy gave the
> permission to access (or deny).
>
>
>
> However, in your case, since it was denied, there might have been no
> policy, but the resource field should have given the resource name as
> “root.test”. If not, we should look into this.
>
>
>
> Any suggestions to improve is welcomed…
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
>
>
>
>
> *From: *Loïc Chanel <lo...@telecomnancy.net>
> *Reply-To: *<us...@ranger.incubator.apache.org>
> *Date: *Tuesday, September 6, 2016 at 8:51 AM
> *To: *<us...@ranger.incubator.apache.org>
> *Subject: *Re: User running job in forbidden queue
>
>
>
> And now I feel like a complete idiot because my actual problem was the
> fact that in Ranger policies I wrote "test" instead of "root.test".
>
> Sorry for the spam, then.
>
>
>
> Regards,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
> 2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Actually, I ran some further tests and the property ranger.add-yarn-authorization
> set to false in ranger-yarn-security seems to prevent anyone to run jobs in
> any queue as my user "test" cannot submit a job into "test" queue according
> to YARN.
>
> Anyone encountered the same issue ?
>
>
>
> FYI, I am using an HDP 2.4 stack.
>
>
>
> Regards,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
> 2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Hi all,
>
>
>
> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
> quite a big issue.
>
> Using Ranger to handle which user can submit job to which queue I
> authorized user "test" to submit jobs on queue "test" only - with the
> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
>
> But even with these settings when user "test" submit a job it goes in the
> "default" queue - to which he shouldn't be able to submit jobs.
>
>
>
> Do you see what I miss here ?
>
> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>
>
>
> Thanks in advance for your inputs,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
>
>
>
>
Re: User running job in forbidden queue
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Loïc
> A possible improvement would be to prevent a user from creating a policy for a queue that doesn't exist.
> Tell me if it sounds feasible, and I'll create the corresponding Jira.
I am not sure this will work. One of Ranger flexible feature is to be able to create policies on resources which are not yet created. We could give validation options to verify if the resource is present or not. This could help in debugging.
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Wednesday, September 7, 2016 at 1:07 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: User running job in forbidden queue
Bosco,
I think the audit log would have been a great indicator of my mistake, but another problem I have to work on is the fact that I haven't any audit logs but the ones in the Plugins tab ;-)
This is what happen when you do not build the cluster yourself ;-)
A possible improvement would be to prevent a user from creating a policy for a queue that doesn't exist.
Tell me if it sounds feasible, and I'll create the corresponding Jira.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 18:40 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
Hi Loïc
Just curious, was the audit log helpful? I understand, this could be frustrating, so during the last couple of releases, in the audit logs, we have added more information to help admins understand which policy gave the permission to access (or deny).
However, in your case, since it was denied, there might have been no policy, but the resource field should have given the resource name as “root.test”. If not, we should look into this.
Any suggestions to improve is welcomed…
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, September 6, 2016 at 8:51 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: User running job in forbidden queue
And now I feel like a complete idiot because my actual problem was the fact that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Actually, I ran some further tests and the property ranger.add-yarn-authorization set to false in ranger-yarn-security seems to prevent anyone to run jobs in any queue as my user "test" cannot submit a job into "test" queue according to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I quite a big issue.
Using Ranger to handle which user can submit job to which queue I authorized user "test" to submit jobs on queue "test" only - with the property ranger.add-yarn-authorization set to false in ranger-yarn-security.
But even with these settings when user "test" submit a job it goes in the "default" queue - to which he shouldn't be able to submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
I will dig into that option, as it sounds what I missed. Thank you Ramesh.
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-07 21:15 GMT+02:00 Ramesh Mani <rm...@hortonworks.com>:
> Loïc,
>
> The lookup functionality in the ranger policy creation will allow you to
> select only the queues which are present in the cluster. If the
> configurations in ranger yarn service is correct this should work as
> expected and also serve as another alternative for this issue.
>
> Thanks,
> Ramesh
>
> From: Loïc Chanel <lo...@telecomnancy.net>
> Reply-To: "user@ranger.incubator.apache.org" <
> user@ranger.incubator.apache.org>
> Date: Wednesday, September 7, 2016 at 1:07 AM
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
>
> Subject: Re: User running job in forbidden queue
>
> Bosco,
>
> I think the audit log would have been a great indicator of my mistake, but
> another problem I have to work on is the fact that I haven't any audit logs
> but the ones in the Plugins tab ;-)
> This is what happen when you do not build the cluster yourself ;-)
>
> A possible improvement would be to prevent a user from creating a policy
> for a queue that doesn't exist.
> Tell me if it sounds feasible, and I'll create the corresponding Jira.
>
> Regards,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
> 2016-09-06 18:40 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
>
>> Hi Loïc
>>
>>
>>
>> Just curious, was the audit log helpful? I understand, this could be
>> frustrating, so during the last couple of releases, in the audit logs, we
>> have added more information to help admins understand which policy gave the
>> permission to access (or deny).
>>
>>
>>
>> However, in your case, since it was denied, there might have been no
>> policy, but the resource field should have given the resource name as
>> “root.test”. If not, we should look into this.
>>
>>
>>
>> Any suggestions to improve is welcomed…
>>
>>
>>
>> Thanks
>>
>>
>>
>> Bosco
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Loïc Chanel <lo...@telecomnancy.net>
>> *Reply-To: *<us...@ranger.incubator.apache.org>
>> *Date: *Tuesday, September 6, 2016 at 8:51 AM
>> *To: *<us...@ranger.incubator.apache.org>
>> *Subject: *Re: User running job in forbidden queue
>>
>>
>>
>> And now I feel like a complete idiot because my actual problem was the
>> fact that in Ranger policies I wrote "test" instead of "root.test".
>>
>> Sorry for the spam, then.
>>
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>> Loïc
>>
>>
>> Loïc CHANEL
>> System Big Data engineer
>> MS&T - WASABI - Worldline (Villeurbanne, France)
>>
>>
>>
>> 2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>
>> Actually, I ran some further tests and the property ranger.add-yarn-authorization
>> set to false in ranger-yarn-security seems to prevent anyone to run jobs in
>> any queue as my user "test" cannot submit a job into "test" queue according
>> to YARN.
>>
>> Anyone encountered the same issue ?
>>
>>
>>
>> FYI, I am using an HDP 2.4 stack.
>>
>>
>>
>> Regards,
>>
>>
>>
>>
>>
>> Loïc
>>
>>
>> Loïc CHANEL
>> System Big Data engineer
>> MS&T - WASABI - Worldline (Villeurbanne, France)
>>
>>
>>
>> 2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>
>> Hi all,
>>
>>
>>
>> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
>> quite a big issue.
>>
>> Using Ranger to handle which user can submit job to which queue I
>> authorized user "test" to submit jobs on queue "test" only - with the
>> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
>>
>> But even with these settings when user "test" submit a job it goes in the
>> "default" queue - to which he shouldn't be able to submit jobs.
>>
>>
>>
>> Do you see what I miss here ?
>>
>> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>>
>>
>>
>> Thanks in advance for your inputs,
>>
>>
>>
>>
>>
>> Loïc
>>
>>
>> Loïc CHANEL
>> System Big Data engineer
>> MS&T - WASABI - Worldline (Villeurbanne, France)
>>
>>
>>
>>
>>
>
>
Re: User running job in forbidden queue
Posted by Ramesh Mani <rm...@hortonworks.com>.
Loïc,
The lookup functionality in the ranger policy creation will allow you to select only the queues which are present in the cluster. If the configurations in ranger yarn service is correct this should work as expected and also serve as another alternative for this issue.
Thanks,
Ramesh
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Date: Wednesday, September 7, 2016 at 1:07 AM
To: "user@ranger.incubator.apache.org<ma...@ranger.incubator.apache.org>" <us...@ranger.incubator.apache.org>>
Subject: Re: User running job in forbidden queue
Bosco,
I think the audit log would have been a great indicator of my mistake, but another problem I have to work on is the fact that I haven't any audit logs but the ones in the Plugins tab ;-)
This is what happen when you do not build the cluster yourself ;-)
A possible improvement would be to prevent a user from creating a policy for a queue that doesn't exist.
Tell me if it sounds feasible, and I'll create the corresponding Jira.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 18:40 GMT+02:00 Don Bosco Durai <bo...@apache.org>>:
Hi Loïc
Just curious, was the audit log helpful? I understand, this could be frustrating, so during the last couple of releases, in the audit logs, we have added more information to help admins understand which policy gave the permission to access (or deny).
However, in your case, since it was denied, there might have been no policy, but the resource field should have given the resource name as "root.test". If not, we should look into this.
Any suggestions to improve is welcomed...
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>>
Reply-To: <us...@ranger.incubator.apache.org>>
Date: Tuesday, September 6, 2016 at 8:51 AM
To: <us...@ranger.incubator.apache.org>>
Subject: Re: User running job in forbidden queue
And now I feel like a complete idiot because my actual problem was the fact that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>>:
Actually, I ran some further tests and the property ranger.add-yarn-authorization set to false in ranger-yarn-security seems to prevent anyone to run jobs in any queue as my user "test" cannot submit a job into "test" queue according to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>>:
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I quite a big issue.
Using Ranger to handle which user can submit job to which queue I authorized user "test" to submit jobs on queue "test" only - with the property ranger.add-yarn-authorization set to false in ranger-yarn-security.
But even with these settings when user "test" submit a job it goes in the "default" queue - to which he shouldn't be able to submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Bosco,
I think the audit log would have been a great indicator of my mistake, but
another problem I have to work on is the fact that I haven't any audit logs
but the ones in the Plugins tab ;-)
This is what happen when you do not build the cluster yourself ;-)
A possible improvement would be to prevent a user from creating a policy
for a queue that doesn't exist.
Tell me if it sounds feasible, and I'll create the corresponding Jira.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 18:40 GMT+02:00 Don Bosco Durai <bo...@apache.org>:
> Hi Loïc
>
>
>
> Just curious, was the audit log helpful? I understand, this could be
> frustrating, so during the last couple of releases, in the audit logs, we
> have added more information to help admins understand which policy gave the
> permission to access (or deny).
>
>
>
> However, in your case, since it was denied, there might have been no
> policy, but the resource field should have given the resource name as
> “root.test”. If not, we should look into this.
>
>
>
> Any suggestions to improve is welcomed…
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
>
>
>
>
> *From: *Loïc Chanel <lo...@telecomnancy.net>
> *Reply-To: *<us...@ranger.incubator.apache.org>
> *Date: *Tuesday, September 6, 2016 at 8:51 AM
> *To: *<us...@ranger.incubator.apache.org>
> *Subject: *Re: User running job in forbidden queue
>
>
>
> And now I feel like a complete idiot because my actual problem was the
> fact that in Ranger policies I wrote "test" instead of "root.test".
>
> Sorry for the spam, then.
>
>
>
> Regards,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
> 2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Actually, I ran some further tests and the property ranger.add-yarn-authorization
> set to false in ranger-yarn-security seems to prevent anyone to run jobs in
> any queue as my user "test" cannot submit a job into "test" queue according
> to YARN.
>
> Anyone encountered the same issue ?
>
>
>
> FYI, I am using an HDP 2.4 stack.
>
>
>
> Regards,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
> 2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> Hi all,
>
>
>
> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
> quite a big issue.
>
> Using Ranger to handle which user can submit job to which queue I
> authorized user "test" to submit jobs on queue "test" only - with the
> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
>
> But even with these settings when user "test" submit a job it goes in the
> "default" queue - to which he shouldn't be able to submit jobs.
>
>
>
> Do you see what I miss here ?
>
> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>
>
>
> Thanks in advance for your inputs,
>
>
>
>
>
> Loïc
>
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
>
>
>
>
Re: User running job in forbidden queue
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Loïc
Just curious, was the audit log helpful? I understand, this could be frustrating, so during the last couple of releases, in the audit logs, we have added more information to help admins understand which policy gave the permission to access (or deny).
However, in your case, since it was denied, there might have been no policy, but the resource field should have given the resource name as “root.test”. If not, we should look into this.
Any suggestions to improve is welcomed…
Thanks
Bosco
From: Loïc Chanel <lo...@telecomnancy.net>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Tuesday, September 6, 2016 at 8:51 AM
To: <us...@ranger.incubator.apache.org>
Subject: Re: User running job in forbidden queue
And now I feel like a complete idiot because my actual problem was the fact that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Actually, I ran some further tests and the property ranger.add-yarn-authorization set to false in ranger-yarn-security seems to prevent anyone to run jobs in any queue as my user "test" cannot submit a job into "test" queue according to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
Hi all,
I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I quite a big issue.
Using Ranger to handle which user can submit job to which queue I authorized user "test" to submit jobs on queue "test" only - with the property ranger.add-yarn-authorization set to false in ranger-yarn-security.
But even with these settings when user "test" submit a job it goes in the "default" queue - to which he shouldn't be able to submit jobs.
Do you see what I miss here ?
If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
Thanks in advance for your inputs,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
And now I feel like a complete idiot because my actual problem was the fact
that in Ranger policies I wrote "test" instead of "root.test".
Sorry for the spam, then.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 11:22 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Actually, I ran some further tests and the property ranger.add-yarn-authorization
> set to false in ranger-yarn-security seems to prevent anyone to run jobs in
> any queue as my user "test" cannot submit a job into "test" queue according
> to YARN.
> Anyone encountered the same issue ?
>
> FYI, I am using an HDP 2.4 stack.
>
> Regards,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>
> 2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
>> Hi all,
>>
>> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
>> quite a big issue.
>> Using Ranger to handle which user can submit job to which queue I
>> authorized user "test" to submit jobs on queue "test" only - with the
>> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
>> But even with these settings when user "test" submit a job it goes in the
>> "default" queue - to which he shouldn't be able to submit jobs.
>>
>> Do you see what I miss here ?
>> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>>
>> Thanks in advance for your inputs,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> System Big Data engineer
>> MS&T - WASABI - Worldline (Villeurbanne, France)
>>
>
>
Re: User running job in forbidden queue
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Actually, I ran some further tests and the property
ranger.add-yarn-authorization
set to false in ranger-yarn-security seems to prevent anyone to run jobs in
any queue as my user "test" cannot submit a job into "test" queue according
to YARN.
Anyone encountered the same issue ?
FYI, I am using an HDP 2.4 stack.
Regards,
Loïc
Loïc CHANEL
System Big Data engineer
MS&T - WASABI - Worldline (Villeurbanne, France)
2016-09-06 10:31 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Hi all,
>
> I'm back on the Hadoop & Multi-tenancy topic and as I ran some tests I
> quite a big issue.
> Using Ranger to handle which user can submit job to which queue I
> authorized user "test" to submit jobs on queue "test" only - with the
> property ranger.add-yarn-authorization set to false in ranger-yarn-security.
> But even with these settings when user "test" submit a job it goes in the
> "default" queue - to which he shouldn't be able to submit jobs.
>
> Do you see what I miss here ?
> If not, do anyone knows how to turn on YARN Ranger plugin debug logs ?
>
> Thanks in advance for your inputs,
>
>
> Loïc
>
> Loïc CHANEL
> System Big Data engineer
> MS&T - WASABI - Worldline (Villeurbanne, France)
>