You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2019/08/07 13:15:02 UTC
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
(Updated Aug. 7, 2019, 1:15 p.m.)
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressed first review comment provided in https://reviews.apache.org/r/71236
Summary (updated)
-----------------
RANGER-2499: (solution-2)Improve bulk create/update policy performance
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
Diff: https://reviews.apache.org/r/71237/diff/3/
Changes: https://reviews.apache.org/r/71237/diff/2-3/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217114
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
Lines 1474 (patched)
<https://reviews.apache.org/r/71237/#comment304374>
Formatting, spaces around operator '!='.
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 3055 (patched)
<https://reviews.apache.org/r/71237/#comment304375>
Formatting.
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Lines 2123 (patched)
<https://reviews.apache.org/r/71237/#comment304376>
Consider using perf logger to keep track of performance of critical code segments.
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Line 4165 (original), 4185 (patched)
<https://reviews.apache.org/r/71237/#comment304377>
Formatting
security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java
Lines 29 (patched)
<https://reviews.apache.org/r/71237/#comment304373>
Formatting. spaces around '='.
- Abhay Kulkarni
On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2019, 1:15 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
> security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/4/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217131
-----------------------------------------------------------
Ship it!
Ship It!
- Abhay Kulkarni
On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2019, 1:15 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
> security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/5/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>