You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2019/08/05 07:03:39 UTC
Review Request 71237: RANGER-2499: Improve bulk create/update policy
performance
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
Diff: https://reviews.apache.org/r/71237/diff/1/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217114
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
Lines 1474 (patched)
<https://reviews.apache.org/r/71237/#comment304374>
Formatting, spaces around operator '!='.
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 3055 (patched)
<https://reviews.apache.org/r/71237/#comment304375>
Formatting.
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Lines 2123 (patched)
<https://reviews.apache.org/r/71237/#comment304376>
Consider using perf logger to keep track of performance of critical code segments.
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Line 4165 (original), 4185 (patched)
<https://reviews.apache.org/r/71237/#comment304377>
Formatting
security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java
Lines 29 (patched)
<https://reviews.apache.org/r/71237/#comment304373>
Formatting. spaces around '='.
- Abhay Kulkarni
On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2019, 1:15 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
> security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/4/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217131
-----------------------------------------------------------
Ship it!
Ship It!
- Abhay Kulkarni
On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2019, 1:15 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
> security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/5/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk
create/update policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
(Updated Aug. 7, 2019, 1:15 p.m.)
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressed first review comment provided in https://reviews.apache.org/r/71236
Summary (updated)
-----------------
RANGER-2499: (solution-2)Improve bulk create/update policy performance
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4
security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
Diff: https://reviews.apache.org/r/71237/diff/3/
Changes: https://reviews.apache.org/r/71237/diff/2-3/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
Re: Review Request 71237: RANGER-2499: Improve bulk create/update
policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
(Updated Aug. 6, 2019, 3:29 a.m.)
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
Diff: https://reviews.apache.org/r/71237/diff/2/
Changes: https://reviews.apache.org/r/71237/diff/1-2/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
Re: Review Request 71237: RANGER-2499: Improve bulk create/update
policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Aug. 5, 2019, 6:01 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
> > Line 4178 (original), 4189 (patched)
> > <https://reviews.apache.org/r/71237/diff/1/?file=2159811#file2159811line4191>
> >
> > totalDeletedPolicies is not incremented in the loop. It will cause bulkModeOnlyFlushAndClear() to be called for each deleted policy. Please review.
Thanks for the review. Still testing this issue with large number of policy resources(like 500 table+columns in each policy). Found performance issue there. Seems we need to incorporate your suggestion mentioned in the Approach-1 if not then little more optimization in Approach-1 patch.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217080
-----------------------------------------------------------
On Aug. 6, 2019, 3:29 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2019, 3:29 a.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/2/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71237: RANGER-2499: Improve bulk create/update
policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217080
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Line 4178 (original), 4189 (patched)
<https://reviews.apache.org/r/71237/#comment304343>
totalDeletedPolicies is not incremented in the loop. It will cause bulkModeOnlyFlushAndClear() to be called for each deleted policy. Please review.
- Abhay Kulkarni
On Aug. 5, 2019, 7:08 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
>
> (Updated Aug. 5, 2019, 7:08 a.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
>
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are:
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
>
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
>
>
> Diff: https://reviews.apache.org/r/71237/diff/1/
>
>
> Testing
> -------
>
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
>
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
>
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71237: RANGER-2499: Improve bulk create/update
policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
(Updated Aug. 5, 2019, 7:08 a.m.)
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
This RR contains alternate solution of RANGER-2499(First RR: https://reviews.apache.org/r/71236)
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description (updated)
-------
**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40
Diff: https://reviews.apache.org/r/71237/diff/1/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal