You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Andrew Luo <an...@outlook.com> on 2023/02/08 16:54:46 UTC
Re: Review Request 74245: RANGER-3899:Policy creation takes more time when there are more users, groups, roles
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74245/#review225167
-----------------------------------------------------------
I'm not a committer but I was the author of the original PR a few years back - happy to see this come back and hope it can be merged soon!
- Andrew Luo
On Jan. 4, 2023, 7:14 a.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74245/
> -----------------------------------------------------------
>
> (Updated Jan. 4, 2023, 7:14 a.m.)
>
>
> Review request for ranger, Don Bosco Durai, Kirby Zhou, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3899
> https://issues.apache.org/jira/browse/RANGER-3899
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Analysis is when policy has 500 users with 5 policy items.
>
> we perform total 526 DB read calls and 504 DB write, 3 DB Batch write calls. This is the reason for policy creation time where 1000 plus DB calls are done sequentially.
> Time is taken to create the policy :4~8 seconds
>
> 1st#Improvement:
>
> After Fix :After changing to batch create of XXPolicyRefUser instead of calling 1 DB write XXPolicyRefUser for every user of the policy:
>
> When Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
>
> we perform total 526 DB read calls and 6 DB write calls, 5 DB Batch write calls.
>
> Time is taken to create the policy :2.6~4 seconds
>
> 2nd#Improvement:
> Getting the ID for each role/group/user one-by-one is slow, especially for large policies with many roles/groups/users. Batching significantly improves performance.
>
> Fixing this part will reduce the DB read calls to a very minimum which will improve performance drastically.
>
> The policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
>
> After fix:
> DB Read count:27
>
> Time taken to create the policy: 1.4 to 2 secs (The policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns))
>
> DB read calls will be reduced to 27 from 526
>
> Time is taken to create the policy :1.4~2 seconds
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java 6cc3509d8
> security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 8ca4f62bb
> security-admin/src/main/java/org/apache/ranger/db/XXRoleDao.java 4e5b692b5
> security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 97bc2680a
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213
>
>
> Diff: https://reviews.apache.org/r/74245/diff/2/
>
>
> Testing
> -------
>
> Tested Created policies via load testing
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>