You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2019/08/04 08:14:19 UTC
Review Request 71236: RANGER-2499: Improve bulk create/update policy
performance
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71236/
-----------------------------------------------------------
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**Problem Statement:** Ranger takes an extremely long time to import bulk policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java 1c48eebcb
security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java 0af7a1da2
security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java 66a57289f
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java 7df957973
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java 1a1b56e4c
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java dce3b0b69
security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java 2bf5ee6c9
security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java f9ea26a31
security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java a7e19bf03
Diff: https://reviews.apache.org/r/71236/diff/1/
Testing
-------
**Performance(With patch):**
Time taken to import 500 policies : 84 Seconds
Time taken to import 1000 policies : 111 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
Re: Review Request 71236: RANGER-2499: Improve bulk create/update
policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Aug. 4, 2019, 7:43 p.m., Abhay Kulkarni wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java
> > Line 68 (original), 68 (patched)
> > <https://reviews.apache.org/r/71236/diff/1/?file=2159525#file2159525line68>
> >
> > Please consider an alternative design where the value of boolean 'flush' - which controlls if JPA cache needs to be flushed - may be initiatized in the thread context. In this design, code changes are minimized. Also, it will make comparing performance with and without the fix easier, if such initialization also may be made dependent on value of some configuration parameter.
I have changed the createpolicy signature in the ServiceREST class where the API can take addition parameter for flush. default value will be true which is same as current behavior.
do you mean to use ThreadContext or ThreadLocal implementation here ?
> On Aug. 4, 2019, 7:43 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
> > Lines 2748 (patched)
> > <https://reviews.apache.org/r/71236/diff/1/?file=2159536#file2159536line2749>
> >
> > Is calling flush after 10 policy deletes/updates/creates somewhat arbitrary? Can this be controlled through some config parameter?
yes 10 is an arbitrary number and we can add this as a config.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71236/#review217066
-----------------------------------------------------------
On Aug. 4, 2019, 8:14 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71236/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2019, 8:14 a.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java 1c48eebcb
> security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java 0af7a1da2
> security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java 66a57289f
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java 7df957973
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java 1a1b56e4c
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java dce3b0b69
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java 2bf5ee6c9
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java f9ea26a31
> security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java a7e19bf03
>
>
> Diff: https://reviews.apache.org/r/71236/diff/1/
>
>
> Testing
> -------
>
> **Performance(With patch):**
>
> Time taken to import 500 policies : 84 Seconds
> Time taken to import 1000 policies : 111 Seconds
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71236: RANGER-2499: Improve bulk create/update
policy performance
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71236/#review217066
-----------------------------------------------------------
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java
Line 68 (original), 68 (patched)
<https://reviews.apache.org/r/71236/#comment304316>
Please consider an alternative design where the value of boolean 'flush' - which controlls if JPA cache needs to be flushed - may be initiatized in the thread context. In this design, code changes are minimized. Also, it will make comparing performance with and without the fix easier, if such initialization also may be made dependent on value of some configuration parameter.
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
Lines 2748 (patched)
<https://reviews.apache.org/r/71236/#comment304315>
Is calling flush after 10 policy deletes/updates/creates somewhat arbitrary? Can this be controlled through some config parameter?
- Abhay Kulkarni
On Aug. 4, 2019, 8:14 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71236/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2019, 8:14 a.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java 1c48eebcb
> security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java 0af7a1da2
> security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java 66a57289f
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java 7df957973
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java 1a1b56e4c
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java dce3b0b69
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java 2bf5ee6c9
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java f9ea26a31
> security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java a7e19bf03
>
>
> Diff: https://reviews.apache.org/r/71236/diff/1/
>
>
> Testing
> -------
>
> **Performance(With patch):**
>
> Time taken to import 500 policies : 84 Seconds
> Time taken to import 1000 policies : 111 Seconds
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 71236: RANGER-2499: Improve bulk create/update
policy performance
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71236/#review217065
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Line 2126 (original)
<https://reviews.apache.org/r/71236/#comment304314>
Reverting changes done in RANGER-2493 (to fix bulk delete policy issue) RR link: https://reviews.apache.org/r/71007/
- Pradeep Agrawal
On Aug. 4, 2019, 8:14 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71236/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2019, 8:14 a.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2499
> https://issues.apache.org/jira/browse/RANGER-2499
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement:** Ranger takes an extremely long time to import bulk policies.
> **Current performance(Without patch):**
> Time taken to import 500 policies : 27+ Minutes
> Time taken to import 1000 policies : 81+ Minutes
>
> **Proposed Solution:**
> in the bulk policy creation/deletion operation proposed patch shall call EntityManager.flush() and EntityManager.clear() after processing a batch of 10 records.
>
> **References: **
> https://en.wikibooks.org/wiki/Java_Persistence/Persisting
> https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java d4879766f
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java e242d9092
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4815f5bdb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ef22354de
> security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 4f339087d
> security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java 1c48eebcb
> security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java 0af7a1da2
> security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java 66a57289f
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java 7df957973
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 348d072e6
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java de5d34307
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java 1a1b56e4c
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java b6f13f47e
> security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java dce3b0b69
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java 2bf5ee6c9
> security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java f9ea26a31
> security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java a7e19bf03
>
>
> Diff: https://reviews.apache.org/r/71236/diff/1/
>
>
> Testing
> -------
>
> **Performance(With patch):**
>
> Time taken to import 500 policies : 84 Seconds
> Time taken to import 1000 policies : 111 Seconds
>
> Time taken to delete 500 policies : 30 Seconds
>
>
> Thanks,
>
> Pradeep Agrawal
>
>