You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@freestoneinfotech.com> on 2016/06/07 03:16:47 UTC
Review Request 48277: RANGER-997 : Improve Policy Listing performance
on Reports page having more than 1000 policies
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48277/
-----------------------------------------------------------
Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-997
https://issues.apache.org/jira/browse/RANGER-997
Repository: ranger
Description
-------
**Problem Statement:** It takes lot of time to load the Report page if more than 1000+ policies exist in Ranger.
**Proposed Soluton:**
Proposed solution usage existing policy cache(RangerServicePoliciesCache.java) available for download policy request and remove implementation of sending sql call to db.
Added searchRangerPolicies(filter) method in ServiceDBStore class which is same as searchRangerPolicies(filter) method of RangerPolicyService class; but policyService.searchRangerPolicies(filter) implemention loads policy details from DB while ServiceDBStore.searchRangerPolicies(filter) calls getServicePolicies(serviceId,filter) method of ServiceDBStore to load the policy detail available in RangerServicePoliciesCache.
Diffs
-----
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java a659d45
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 2d5440d
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java a3cd613
Diff: https://reviews.apache.org/r/48277/diff/
Testing
-------
***Steps Performed(Without patch):***
1. Installed and Started Ranger without patch and created 3000 polices under HDFS service.
2. Logged into Ranger Admin and clicked on 'Report page'.
3. Report page loading took approx 13 minute to process the result at server end and few more minutes to render the result on Ranger UI.
4. Retried previous step three more times and processing time in each attempt was approx 12-13 minute.
***Steps Performed(With patch):***
1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
2. After Starting Ranger; logged into Ranger Admin and clicked on 'Report page'.
**Expected Behaviour:** Report page processing at server end should take less than 13 minutes at server end.
**Actual Behavior:** Report page processing at server end took only 4 seconds at server end and few more seconds to render the result on Ranger UI.
Note : Retried the same three more times and processing time at server end in each attempt was ~4 seconds.
Thanks,
Pradeep Agrawal
Re: Review Request 48277: RANGER-997 : Improve Policy Listing
performance on Reports page having more than 1000 policies
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48277/#review136488
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On June 7, 2016, 2:37 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48277/
> -----------------------------------------------------------
>
> (Updated June 7, 2016, 2:37 p.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-997
> https://issues.apache.org/jira/browse/RANGER-997
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement:** It takes lot of time to load the Report page if more than 1000+ policies exist in Ranger.
>
> **Proposed Soluton:**
> Proposed solution usage existing policy cache(RangerServicePoliciesCache.java) available for download policy request and remove implementation of sending sql call to db.
> Added searchRangerPolicies(filter) method in ServiceDBStore class which is same as searchRangerPolicies(filter) method of RangerPolicyService class; but policyService.searchRangerPolicies(filter) implemention loads policy details from DB while ServiceDBStore.searchRangerPolicies(filter) calls getServicePolicies(serviceId,filter) method of ServiceDBStore to load the policy detail available in RangerServicePoliciesCache.
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java a659d45
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 2d5440d
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java bde18bd
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java a3cd613
> security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyServiceBase.java 75e2ebd
>
> Diff: https://reviews.apache.org/r/48277/diff/
>
>
> Testing
> -------
>
> ***Steps Performed(Without patch):***
> 1. Installed and Started Ranger without patch and created 3000 polices under HDFS service.
> 2. Logged into Ranger Admin and clicked on 'Report page'.
> 3. Report page loading took approx 13 minute to process the result at server end and few more minutes to render the result on Ranger UI.
> 4. Retried previous step three more times and processing time in each attempt was approx 12-13 minute.
>
> ***Steps Performed(With patch):***
> 1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
> 2. After Starting Ranger; logged into Ranger Admin and clicked on 'Report page'.
>
> **Expected Behaviour:** Report page processing at server end should take less than 13 minutes at server end.
>
> **Actual Behavior:** Report page processing at server end took only 4 seconds at server end and few more seconds to render the result on Ranger UI.
>
> Note : Retried the same three more times and processing time at server end in each attempt was ~4 seconds.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 48277: RANGER-997 : Improve Policy Listing
performance on Reports page having more than 1000 policies
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48277/
-----------------------------------------------------------
(Updated June 7, 2016, 2:37 p.m.)
Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Addressed review comments; Removed RangerPolicyServiceBase.searchRangerPolicies() method and related junit test; Tested in local environment.
Bugs: RANGER-997
https://issues.apache.org/jira/browse/RANGER-997
Repository: ranger
Description
-------
**Problem Statement:** It takes lot of time to load the Report page if more than 1000+ policies exist in Ranger.
**Proposed Soluton:**
Proposed solution usage existing policy cache(RangerServicePoliciesCache.java) available for download policy request and remove implementation of sending sql call to db.
Added searchRangerPolicies(filter) method in ServiceDBStore class which is same as searchRangerPolicies(filter) method of RangerPolicyService class; but policyService.searchRangerPolicies(filter) implemention loads policy details from DB while ServiceDBStore.searchRangerPolicies(filter) calls getServicePolicies(serviceId,filter) method of ServiceDBStore to load the policy detail available in RangerServicePoliciesCache.
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java a659d45
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 2d5440d
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java bde18bd
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java a3cd613
security-admin/src/test/java/org/apache/ranger/service/TestRangerPolicyServiceBase.java 75e2ebd
Diff: https://reviews.apache.org/r/48277/diff/
Testing
-------
***Steps Performed(Without patch):***
1. Installed and Started Ranger without patch and created 3000 polices under HDFS service.
2. Logged into Ranger Admin and clicked on 'Report page'.
3. Report page loading took approx 13 minute to process the result at server end and few more minutes to render the result on Ranger UI.
4. Retried previous step three more times and processing time in each attempt was approx 12-13 minute.
***Steps Performed(With patch):***
1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
2. After Starting Ranger; logged into Ranger Admin and clicked on 'Report page'.
**Expected Behaviour:** Report page processing at server end should take less than 13 minutes at server end.
**Actual Behavior:** Report page processing at server end took only 4 seconds at server end and few more seconds to render the result on Ranger UI.
Note : Retried the same three more times and processing time at server end in each attempt was ~4 seconds.
Thanks,
Pradeep Agrawal
Re: Review Request 48277: RANGER-997 : Improve Policy Listing
performance on Reports page having more than 1000 policies
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48277/#review136405
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java (line 3636)
<https://reviews.apache.org/r/48277/#comment201450>
If this method replaces RangerPolicyServiceBase.searchRangerPolicies(), consider removing that method.
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java (line 3672)
<https://reviews.apache.org/r/48277/#comment201449>
policyList might already have policy with id=xPol2.getId() - when the user is in multiple groups and if one or more policies have multiple of these groups in its policy items.
Please review and ensure there are no duplicates in policyList.
- Madhan Neethiraj
On June 7, 2016, 3:16 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48277/
> -----------------------------------------------------------
>
> (Updated June 7, 2016, 3:16 a.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-997
> https://issues.apache.org/jira/browse/RANGER-997
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement:** It takes lot of time to load the Report page if more than 1000+ policies exist in Ranger.
>
> **Proposed Soluton:**
> Proposed solution usage existing policy cache(RangerServicePoliciesCache.java) available for download policy request and remove implementation of sending sql call to db.
> Added searchRangerPolicies(filter) method in ServiceDBStore class which is same as searchRangerPolicies(filter) method of RangerPolicyService class; but policyService.searchRangerPolicies(filter) implemention loads policy details from DB while ServiceDBStore.searchRangerPolicies(filter) calls getServicePolicies(serviceId,filter) method of ServiceDBStore to load the policy detail available in RangerServicePoliciesCache.
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java a659d45
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 2d5440d
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java a3cd613
>
> Diff: https://reviews.apache.org/r/48277/diff/
>
>
> Testing
> -------
>
> ***Steps Performed(Without patch):***
> 1. Installed and Started Ranger without patch and created 3000 polices under HDFS service.
> 2. Logged into Ranger Admin and clicked on 'Report page'.
> 3. Report page loading took approx 13 minute to process the result at server end and few more minutes to render the result on Ranger UI.
> 4. Retried previous step three more times and processing time in each attempt was approx 12-13 minute.
>
> ***Steps Performed(With patch):***
> 1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
> 2. After Starting Ranger; logged into Ranger Admin and clicked on 'Report page'.
>
> **Expected Behaviour:** Report page processing at server end should take less than 13 minutes at server end.
>
> **Actual Behavior:** Report page processing at server end took only 4 seconds at server end and few more seconds to render the result on Ranger UI.
>
> Note : Retried the same three more times and processing time at server end in each attempt was ~4 seconds.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>