You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2019/08/05 07:03:39 UTC

Review Request 71237: RANGER-2499: Improve bulk create/update policy performance

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------

Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Bugs: RANGER-2499
    https://issues.apache.org/jira/browse/RANGER-2499


Repository: ranger


Description
-------

**Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes

**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.


** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are: 
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.


**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/


Diffs
-----

  agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
  security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
  security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
  security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
  security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 


Diff: https://reviews.apache.org/r/71237/diff/1/


Testing
-------

Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds

Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds


Time taken to delete 500 policies : 30 Seconds


Thanks,

Pradeep Agrawal


Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk create/update policy performance

Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217114
-----------------------------------------------------------




security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
Lines 1474 (patched)
<https://reviews.apache.org/r/71237/#comment304374>

    Formatting, spaces around operator '!='.



security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 3055 (patched)
<https://reviews.apache.org/r/71237/#comment304375>

    Formatting.



security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Lines 2123 (patched)
<https://reviews.apache.org/r/71237/#comment304376>

    Consider using perf logger to keep track of performance of critical code segments.



security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Line 4165 (original), 4185 (patched)
<https://reviews.apache.org/r/71237/#comment304377>

    Formatting



security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java
Lines 29 (patched)
<https://reviews.apache.org/r/71237/#comment304373>

    Formatting. spaces around '='.


- Abhay Kulkarni


On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
> 
> (Updated Aug. 7, 2019, 1:15 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2499
>     https://issues.apache.org/jira/browse/RANGER-2499
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
> 
> **Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
> 
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
> 
> 
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are: 
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
> 
> 
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
>   security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
>   security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
>   security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4 
>   security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f 
>   security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
>   security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655 
>   security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 
> 
> 
> Diff: https://reviews.apache.org/r/71237/diff/4/
> 
> 
> Testing
> -------
> 
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
> 
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
> 
> 
> Time taken to delete 500 policies : 30 Seconds
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>


Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk create/update policy performance

Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217131
-----------------------------------------------------------


Ship it!




Ship It!

- Abhay Kulkarni


On Aug. 7, 2019, 1:15 p.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
> 
> (Updated Aug. 7, 2019, 1:15 p.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2499
>     https://issues.apache.org/jira/browse/RANGER-2499
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
> 
> **Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
> 
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
> 
> 
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are: 
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
> 
> 
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
>   security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
>   security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
>   security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4 
>   security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f 
>   security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
>   security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655 
>   security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 
> 
> 
> Diff: https://reviews.apache.org/r/71237/diff/5/
> 
> 
> Testing
> -------
> 
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
> 
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
> 
> 
> Time taken to delete 500 policies : 30 Seconds
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>


Re: Review Request 71237: RANGER-2499: (solution-2)Improve bulk create/update policy performance

Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------

(Updated Aug. 7, 2019, 1:15 p.m.)


Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
-------

Addressed first review comment provided in https://reviews.apache.org/r/71236


Summary (updated)
-----------------

RANGER-2499: (solution-2)Improve bulk create/update policy performance


Bugs: RANGER-2499
    https://issues.apache.org/jira/browse/RANGER-2499


Repository: ranger


Description
-------

**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**

**Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes

**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.


** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are: 
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.


**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/


Diffs (updated)
-----

  agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
  security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
  security-admin/src/main/java/org/apache/ranger/common/ContextUtil.java 9554b4f53 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
  security-admin/src/main/java/org/apache/ranger/security/context/RangerAdminOpContext.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/security/context/RangerContextHolder.java 9884163e4 
  security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSecurityContextFormationFilter.java d75b9032f 
  security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
  security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4ea2ad655 
  security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 


Diff: https://reviews.apache.org/r/71237/diff/3/

Changes: https://reviews.apache.org/r/71237/diff/2-3/


Testing
-------

Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds

Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds


Time taken to delete 500 policies : 30 Seconds


Thanks,

Pradeep Agrawal


Re: Review Request 71237: RANGER-2499: Improve bulk create/update policy performance

Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------

(Updated Aug. 6, 2019, 3:29 a.m.)


Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
-------

Addressed review comments


Bugs: RANGER-2499
    https://issues.apache.org/jira/browse/RANGER-2499


Repository: ranger


Description
-------

**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**

**Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes

**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.


** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are: 
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.


**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/


Diffs (updated)
-----

  agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
  security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
  security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
  security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
  security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 


Diff: https://reviews.apache.org/r/71237/diff/2/

Changes: https://reviews.apache.org/r/71237/diff/1-2/


Testing
-------

Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds

Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds


Time taken to delete 500 policies : 30 Seconds


Thanks,

Pradeep Agrawal


Re: Review Request 71237: RANGER-2499: Improve bulk create/update policy performance

Posted by Pradeep Agrawal <pr...@gmail.com>.

> On Aug. 5, 2019, 6:01 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
> > Line 4178 (original), 4189 (patched)
> > <https://reviews.apache.org/r/71237/diff/1/?file=2159811#file2159811line4191>
> >
> >     totalDeletedPolicies is not incremented in the loop. It will cause bulkModeOnlyFlushAndClear() to be called for each deleted policy. Please review.

Thanks for the review. Still testing this issue with large number of policy resources(like 500 table+columns in each policy). Found performance issue there. Seems we need to incorporate your suggestion mentioned in the Approach-1 if not then little more optimization in Approach-1 patch.


- Pradeep


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217080
-----------------------------------------------------------


On Aug. 6, 2019, 3:29 a.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
> 
> (Updated Aug. 6, 2019, 3:29 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2499
>     https://issues.apache.org/jira/browse/RANGER-2499
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
> 
> **Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
> 
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
> 
> 
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are: 
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
> 
> 
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
>   security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
>   security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
>   security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
>   security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 
> 
> 
> Diff: https://reviews.apache.org/r/71237/diff/2/
> 
> 
> Testing
> -------
> 
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
> 
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
> 
> 
> Time taken to delete 500 policies : 30 Seconds
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>


Re: Review Request 71237: RANGER-2499: Improve bulk create/update policy performance

Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/#review217080
-----------------------------------------------------------




security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Line 4178 (original), 4189 (patched)
<https://reviews.apache.org/r/71237/#comment304343>

    totalDeletedPolicies is not incremented in the loop. It will cause bulkModeOnlyFlushAndClear() to be called for each deleted policy. Please review.


- Abhay Kulkarni


On Aug. 5, 2019, 7:08 a.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71237/
> -----------------------------------------------------------
> 
> (Updated Aug. 5, 2019, 7:08 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2499
>     https://issues.apache.org/jira/browse/RANGER-2499
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**
> 
> **Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
> 
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
> 
> 
> ** Note: **
> Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
> Important changes are: 
> 1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
> 2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
> 3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
> 4) few null checks in the existing code.
> 
> 
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
>   security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
>   security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
>   security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
>   security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 
> 
> 
> Diff: https://reviews.apache.org/r/71237/diff/1/
> 
> 
> Testing
> -------
> 
> Time taken to import 500 policies(500 create) : 45 Seconds
> Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
> 
> Time taken to import 1000 policies(1000 create) : 76 Seconds
> Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
> 
> 
> Time taken to delete 500 policies : 30 Seconds
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>


Re: Review Request 71237: RANGER-2499: Improve bulk create/update policy performance

Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------

(Updated Aug. 5, 2019, 7:08 a.m.)


Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
-------

This RR contains alternate solution of RANGER-2499(First RR: https://reviews.apache.org/r/71236)


Bugs: RANGER-2499
    https://issues.apache.org/jira/browse/RANGER-2499


Repository: ranger


Description (updated)
-------

**This RR contains alternate solution of RANGER-2499. First solution RR is https://reviews.apache.org/r/71236/**

**Problem Statement:**  Ranger takes an extremely long time to import bulk policies. 
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes

**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.


** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am trying to generalize the solution for bulk create and update policies also.
Important changes are: 
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through ranger-admin-default-site.xml file
4) few null checks in the existing code.


**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/


Diffs
-----

  agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092 
  security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java d49ea9886 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java 0788fb5bc 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6 
  security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307 
  security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java 019816e49 
  security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml eb7947a47 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java 655e17f40 


Diff: https://reviews.apache.org/r/71237/diff/1/


Testing
-------

Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds

Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds


Time taken to delete 500 policies : 30 Seconds


Thanks,

Pradeep Agrawal