You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Ramachandran Krishnan <ra...@gmail.com> on 2022/09/28 15:32:16 UTC
Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
-----------------------------------------------------------
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900
Repository: ranger
Description
-------
We used DB batch write call instead of 1 DB write call for every user,role,group deference
Diffs
-----
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
Diff: https://reviews.apache.org/r/74148/diff/1/
Testing
-------
test with 1 Role contains 1k other roles and 1k groups
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds
Thanks,
Ramachandran Krishnan
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
> On Oct. 5, 2022, 9:48 a.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
> > Lines 46 (patched)
> > <https://reviews.apache.org/r/74148/diff/1/?file=2270261#file2270261line46>
> >
> > This change overlaps with the changes in another patch - https://reviews.apache.org/r/74138/, for RANGER-3929. Can you please review and consolidate these two patches? Thanks!
Sure we will consolidate this change with along with https://reviews.apache.org/r/74138 in this patch itself
- Ramachandran
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224746
-----------------------------------------------------------
On Sept. 28, 2022, 3:32 p.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2022, 3:32 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/1/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224746
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
Lines 46 (patched)
<https://reviews.apache.org/r/74148/#comment313524>
This change overlaps with the changes in another patch - https://reviews.apache.org/r/74138/, for RANGER-3929. Can you please review and consolidate these two patches? Thanks!
- Madhan Neethiraj
On Sept. 28, 2022, 3:32 p.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2022, 3:32 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/1/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224757
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java
Line 168 (original), 172 (patched)
<https://reviews.apache.org/r/74148/#comment313533>
With the new findByRoleId() methods returning List<Long>, how about directly assigning to xxRoleRefUserIds, xxRoleRefRoleIds and xxRoleRefGroupByIds - like:
List<Long> xxRoleRefUserIds = xRoleUserDao.findIdsByRoleId(roleId);
List<Long> xxRoleRefRoleIds = xRoleRoleDao.findIdsByRoleId(roleId);
List<Long> xxRoleRefGroupByIds = xRoleGroupDao.findIdsByRoleId(roleId);
- Madhan Neethiraj
On Oct. 6, 2022, 6:09 p.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Oct. 6, 2022, 6:09 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/3/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224775
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Oct. 11, 2022, 3:33 a.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2022, 3:33 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/5/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
-----------------------------------------------------------
(Updated Oct. 11, 2022, 3:33 a.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900
Repository: ranger
Description
-------
We used DB batch write call instead of 1 DB write call for every user,role,group deference
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
Diff: https://reviews.apache.org/r/74148/diff/5/
Changes: https://reviews.apache.org/r/74148/diff/4-5/
Testing
-------
test with 1 Role contains 1k other roles and 1k groups
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds
Thanks,
Ramachandran Krishnan
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
-----------------------------------------------------------
(Updated Oct. 7, 2022, 3:37 a.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900
Repository: ranger
Description
-------
We used DB batch write call instead of 1 DB write call for every user,role,group deference
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
Diff: https://reviews.apache.org/r/74148/diff/4/
Changes: https://reviews.apache.org/r/74148/diff/3-4/
Testing
-------
test with 1 Role contains 1k other roles and 1k groups
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds
Thanks,
Ramachandran Krishnan
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
-----------------------------------------------------------
(Updated Oct. 6, 2022, 6:09 p.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900
Repository: ranger
Description
-------
We used DB batch write call instead of 1 DB write call for every user,role,group deference
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
Diff: https://reviews.apache.org/r/74148/diff/3/
Changes: https://reviews.apache.org/r/74148/diff/2-3/
Testing
-------
test with 1 Role contains 1k other roles and 1k groups
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds
Thanks,
Ramachandran Krishnan
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Madhan Neethiraj <ma...@apache.org>.
> On Oct. 5, 2022, 4:09 p.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
> > Lines 139 (patched)
> > <https://reviews.apache.org/r/74148/diff/2/?file=2270432#file2270432line139>
> >
> > Current behavior is to delete all IDs in one call. To retain this, consider the following:
> >
> > if (BATCH_DELETE_BATCH_SIZE < 0) {
> > getEntityManager().createNamedQuery(namedQuery, tClass)
> > .setParameter(paramName, ids).executeUpdate();
> > } else {
> > // batch deletes
> > }
>
> Ramachandran Krishnan wrote:
> When the list size is above 1000, this flow will not work due to the limitation of inclause values size.
> As far as i know better we keep batch deletes to avoid those kind of errors .
deleting one at a time will retain the behavior for cleanupRefTables(), which currently deletes one entity at a time. However, other use of bulk-delete currently delete all entities in one call; these will be changed to delete one at a time. Hence my suggestion above.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224750
-----------------------------------------------------------
On Oct. 7, 2022, 3:37 a.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Oct. 7, 2022, 3:37 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/4/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
> On Oct. 5, 2022, 4:09 p.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java
> > Line 168 (original), 172 (patched)
> > <https://reviews.apache.org/r/74148/diff/2/?file=2270431#file2270431line172>
> >
> > It will help avoid loading of XXRoleRefUser/XXRoleRefGroup/XXRoleRefRole entities into memory. Instead consider using the approach used in following methods:
> > - XXPolicyRefUserDao.deleteByPolicyId()
> > - XXPolicyRefGroupDao.deleteByPolicyId()
> > - XXPolicyRefRoleDao.deleteByPolicyId()
Yeah good thought .I will add this change to avoid loading of XXRoleRefUser/XXRoleRefGroup/XXRoleRefRole entities into memory instead of we will get the ids in single call
> On Oct. 5, 2022, 4:09 p.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
> > Lines 139 (patched)
> > <https://reviews.apache.org/r/74148/diff/2/?file=2270432#file2270432line139>
> >
> > Current behavior is to delete all IDs in one call. To retain this, consider the following:
> >
> > if (BATCH_DELETE_BATCH_SIZE < 0) {
> > getEntityManager().createNamedQuery(namedQuery, tClass)
> > .setParameter(paramName, ids).executeUpdate();
> > } else {
> > // batch deletes
> > }
When the list size is above 1000, this flow will not work due to the limitation of inclause values size.
As far as i know better we keep batch deletes to avoid those kind of errors .
- Ramachandran
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224750
-----------------------------------------------------------
On Oct. 5, 2022, 3:42 p.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2022, 3:42 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/2/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/#review224750
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java
Line 168 (original), 172 (patched)
<https://reviews.apache.org/r/74148/#comment313529>
It will help avoid loading of XXRoleRefUser/XXRoleRefGroup/XXRoleRefRole entities into memory. Instead consider using the approach used in following methods:
- XXPolicyRefUserDao.deleteByPolicyId()
- XXPolicyRefGroupDao.deleteByPolicyId()
- XXPolicyRefRoleDao.deleteByPolicyId()
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
Lines 139 (patched)
<https://reviews.apache.org/r/74148/#comment313530>
Current behavior is to delete all IDs in one call. To retain this, consider the following:
if (BATCH_DELETE_BATCH_SIZE < 0) {
getEntityManager().createNamedQuery(namedQuery, tClass)
.setParameter(paramName, ids).executeUpdate();
} else {
// batch deletes
}
- Madhan Neethiraj
On Oct. 5, 2022, 3:42 p.m., Ramachandran Krishnan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74148/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2022, 3:42 p.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3900
> https://issues.apache.org/jira/browse/RANGER-3900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> We used DB batch write call instead of 1 DB write call for every user,role,group deference
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
> security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
> security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
>
>
> Diff: https://reviews.apache.org/r/74148/diff/2/
>
>
> Testing
> -------
>
> test with 1 Role contains 1k other roles and 1k groups
> while deleting the role it took 11.577 seconds
> After fix
> it took 1.021 seconds
>
>
> Thanks,
>
> Ramachandran Krishnan
>
>
Re: Review Request 74148: Roles deletion Takes time in Apache Ranger when there are more users, groups, roles
Posted by Ramachandran Krishnan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74148/
-----------------------------------------------------------
(Updated Oct. 5, 2022, 3:42 p.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3900
https://issues.apache.org/jira/browse/RANGER-3900
Repository: ranger
Description
-------
We used DB batch write call instead of 1 DB write call for every user,role,group deference
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/biz/RoleRefUpdater.java 56f7ec4c8
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java fc56ff88b
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java a8233e30c
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java bc17fcdb0
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java df5f7cd94
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java dc41aeadb
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 738c6ff49
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 35433c758
security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java eced7b261
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefGroupDao.java eb470f260
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefRoleDao.java 8f6fc8cfa
security-admin/src/main/java/org/apache/ranger/db/XXRoleRefUserDao.java ba9fb494b
security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0
Diff: https://reviews.apache.org/r/74148/diff/2/
Changes: https://reviews.apache.org/r/74148/diff/1-2/
Testing
-------
test with 1 Role contains 1k other roles and 1k groups
while deleting the role it took 11.577 seconds
After fix
it took 1.021 seconds
Thanks,
Ramachandran Krishnan