You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Madhan Neethiraj <ma...@apache.org> on 2015/11/11 00:47:54 UTC
Review Request 40160: RANGER-702: optimize policy download performance
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40160/
-----------------------------------------------------------
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-702
https://issues.apache.org/jira/browse/RANGER-702
Repository: ranger
Description
-------
Optimize the code that retrieves policies from the database - to minimize the number of queries to the database. This is an update over review 39475 - to handle failure scenarios gracefully.
Added PERF logs to public methods in RangerPolicyRetriever and ServiceREST classes; to enable PERF logs, configure log4j to set INFO level for "ranger.perf.".
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfTracer.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 7dbf9fb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java d9952e0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 4ed59f1
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDao.java 9c3fe2e
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java fe0c806
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java d8235bc
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 0cdb75e
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java ecf4da0
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ff4d297
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 9ecd77e
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 12367e6
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8c73c4c
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyWithAssignedIdService.java 0dae4bd
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4de99e2
security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyService.java dc375f5
Diff: https://reviews.apache.org/r/40160/diff/
Testing
-------
After this updates, it took less than 10 seconds to download ~1500 Hive policies; earlier this took more than 10 minutes to download.
Thanks,
Madhan Neethiraj
Re: Review Request 40160: RANGER-702: optimize policy download
performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40160/#review106146
-----------------------------------------------------------
Ship it!
Ship It!
- Abhay Kulkarni
On Nov. 11, 2015, 6:45 p.m., Madhan Neethiraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40160/
> -----------------------------------------------------------
>
> (Updated Nov. 11, 2015, 6:45 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-702
> https://issues.apache.org/jira/browse/RANGER-702
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Optimize the code that retrieves policies from the database - to minimize the number of queries to the database. This is an update over review 39475 - to handle failure scenarios gracefully.
> Added PERF logs to public methods in RangerPolicyRetriever and ServiceREST classes; to enable PERF logs, configure log4j to set INFO level for "ranger.perf.".
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfTracer.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 7dbf9fb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java d9952e0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 4ed59f1
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDao.java 9c3fe2e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java fe0c806
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java d8235bc
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 0cdb75e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java ecf4da0
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ff4d297
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 9ecd77e
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 12367e6
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8c73c4c
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyWithAssignedIdService.java 0dae4bd
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4de99e2
> security-admin/src/main/webapp/WEB-INF/log4j.xml 22ec74e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyService.java dc375f5
>
> Diff: https://reviews.apache.org/r/40160/diff/
>
>
> Testing
> -------
>
> After this updates, it took less than 10 seconds to download ~1500 Hive policies; earlier this took more than 10 minutes to download.
>
>
> Thanks,
>
> Madhan Neethiraj
>
>
Re: Review Request 40160: RANGER-702: optimize policy download
performance
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40160/
-----------------------------------------------------------
(Updated Nov. 11, 2015, 6:45 p.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Updated Ranger admin log4j.xml to include perf_appender
Bugs: RANGER-702
https://issues.apache.org/jira/browse/RANGER-702
Repository: ranger
Description
-------
Optimize the code that retrieves policies from the database - to minimize the number of queries to the database. This is an update over review 39475 - to handle failure scenarios gracefully.
Added PERF logs to public methods in RangerPolicyRetriever and ServiceREST classes; to enable PERF logs, configure log4j to set INFO level for "ranger.perf.".
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfTracer.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 7dbf9fb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java d9952e0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 4ed59f1
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDao.java 9c3fe2e
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java fe0c806
security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java d8235bc
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 0cdb75e
security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java ecf4da0
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ff4d297
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 9ecd77e
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 12367e6
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8c73c4c
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyWithAssignedIdService.java 0dae4bd
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4de99e2
security-admin/src/main/webapp/WEB-INF/log4j.xml 22ec74e
security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyService.java dc375f5
Diff: https://reviews.apache.org/r/40160/diff/
Testing
-------
After this updates, it took less than 10 seconds to download ~1500 Hive policies; earlier this took more than 10 minutes to download.
Thanks,
Madhan Neethiraj
Re: Review Request 40160: RANGER-702: optimize policy download
performance
Posted by Madhan Neethiraj <ma...@apache.org>.
> On Nov. 11, 2015, 3:16 a.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java, line 630
> > <https://reviews.apache.org/r/40160/diff/1/?file=1122575#file1122575line630>
> >
> > Can there be multiple condition with same condition-type? Can the condition values of multiple conditions with same type be collected together into one RangerPolicyItemCondition object?
>> Can there be multiple condition with same condition-type?
Yes.
>> Can the condition values of multiple conditions with same type be collected together into one RangerPolicyItemCondition object?
Yes. The current code does collect all values of a condition-type into a single RangerPolicyItemCondition.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40160/#review106011
-----------------------------------------------------------
On Nov. 10, 2015, 11:47 p.m., Madhan Neethiraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40160/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 11:47 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-702
> https://issues.apache.org/jira/browse/RANGER-702
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Optimize the code that retrieves policies from the database - to minimize the number of queries to the database. This is an update over review 39475 - to handle failure scenarios gracefully.
> Added PERF logs to public methods in RangerPolicyRetriever and ServiceREST classes; to enable PERF logs, configure log4j to set INFO level for "ranger.perf.".
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfTracer.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 7dbf9fb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java d9952e0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 4ed59f1
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDao.java 9c3fe2e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java fe0c806
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java d8235bc
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 0cdb75e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java ecf4da0
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ff4d297
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 9ecd77e
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 12367e6
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8c73c4c
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyWithAssignedIdService.java 0dae4bd
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4de99e2
> security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyService.java dc375f5
>
> Diff: https://reviews.apache.org/r/40160/diff/
>
>
> Testing
> -------
>
> After this updates, it took less than 10 seconds to download ~1500 Hive policies; earlier this took more than 10 minutes to download.
>
>
> Thanks,
>
> Madhan Neethiraj
>
>
Re: Review Request 40160: RANGER-702: optimize policy download
performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40160/#review106011
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java (line 630)
<https://reviews.apache.org/r/40160/#comment164711>
Can there be multiple condition with same condition-type? Can the condition values of multiple conditions with same type be collected together into one RangerPolicyItemCondition object?
- Abhay Kulkarni
On Nov. 10, 2015, 11:47 p.m., Madhan Neethiraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40160/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2015, 11:47 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-702
> https://issues.apache.org/jira/browse/RANGER-702
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Optimize the code that retrieves policies from the database - to minimize the number of queries to the database. This is an update over review 39475 - to handle failure scenarios gracefully.
> Added PERF logs to public methods in RangerPolicyRetriever and ServiceREST classes; to enable PERF logs, configure log4j to set INFO level for "ranger.perf.".
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfTracer.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 7dbf9fb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java d9952e0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 4ed59f1
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDao.java 9c3fe2e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java fe0c806
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java d8235bc
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 0cdb75e
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java ecf4da0
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ff4d297
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 9ecd77e
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 12367e6
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8c73c4c
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyWithAssignedIdService.java 0dae4bd
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4de99e2
> security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyService.java dc375f5
>
> Diff: https://reviews.apache.org/r/40160/diff/
>
>
> Testing
> -------
>
> After this updates, it took less than 10 seconds to download ~1500 Hive policies; earlier this took more than 10 minutes to download.
>
>
> Thanks,
>
> Madhan Neethiraj
>
>