You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Mahesh Bandal <ma...@gmail.com> on 2021/07/20 13:52:12 UTC
Re: Review Request 73452: RANGER-3023: Permission tab takes longer
time to load with large number of users and group_users data
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73452/
-----------------------------------------------------------
(Updated July 20, 2021, 1:52 p.m.)
Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3023
https://issues.apache.org/jira/browse/RANGER-3023
Repository: ranger
Description
-------
GET API /service/xusers/permission takes longer time to load with following number of users and group mappings in db.
select count(*) from x_user;
109040
select count(*) from x_portal_user;
109038
select count(*) from x_group_users;
689952
Current problem : For every ModuleDef, db call to fetch all XXUser, XXPortalUser and creating a Map object using xUserService.getXXPortalUserIdXXUserMap() is a costly operation. Similarly for xGroupService.getXXGroupIdXXGroupMap().
Solution:
In the following patch, I have overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once.
i.e. Map<Long, XXUser> xXPortalUserIdXXUserMap = xUserService.getXXPortalUserIdXXUserMap();
Map<Long, XXGroup> xXGroupMap = xGroupService.getXXGroupIdXXGroupMap();
These two objects will be passed to an overloaded method populateViewBean()
Diffs
-----
security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java d5ca38548
Diff: https://reviews.apache.org/r/73452/diff/2/
Testing
-------
xUserService.getXXPortalUserIdXXUserMap() takes approximately 2000 milliseconds.
xGroupService.getXXGroupIdXXGroupMap() takes approximately 500 milliseconds.
Before patch, XModuleDefServiceBase.searchModuleDef() took 30252 milliseconds.
After patch, XModuleDefService.searchModuleDef() took 13766 milliseconds.
GET API /service/xusers/permission response improved by ~16 seconds for the above mentioned dataset.
Thanks,
Mahesh Bandal
Re: Review Request 73452: RANGER-3023: Permission tab takes longer
time to load with large number of users and group_users data
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73452/#review223258
-----------------------------------------------------------
Ship it!
Ship It!
- Pradeep Agrawal
On July 20, 2021, 1:52 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73452/
> -----------------------------------------------------------
>
> (Updated July 20, 2021, 1:52 p.m.)
>
>
> Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3023
> https://issues.apache.org/jira/browse/RANGER-3023
>
>
> Repository: ranger
>
>
> Description
> -------
>
> GET API /service/xusers/permission takes longer time to load with following number of users and group mappings in db.
>
> select count(*) from x_user;
> 109040
>
> select count(*) from x_portal_user;
> 109038
>
> select count(*) from x_group_users;
> 689952
>
> Current problem : For every ModuleDef, db call to fetch all XXUser, XXPortalUser and creating a Map object using xUserService.getXXPortalUserIdXXUserMap() is a costly operation. Similarly for xGroupService.getXXGroupIdXXGroupMap().
>
> Solution:
> In the following patch, I have overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once.
> i.e. Map<Long, XXUser> xXPortalUserIdXXUserMap = xUserService.getXXPortalUserIdXXUserMap();
> Map<Long, XXGroup> xXGroupMap = xGroupService.getXXGroupIdXXGroupMap();
>
> These two objects will be passed to an overloaded method populateViewBean()
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java d5ca38548
>
>
> Diff: https://reviews.apache.org/r/73452/diff/2/
>
>
> Testing
> -------
>
> xUserService.getXXPortalUserIdXXUserMap() takes approximately 2000 milliseconds.
> xGroupService.getXXGroupIdXXGroupMap() takes approximately 500 milliseconds.
>
> Before patch, XModuleDefServiceBase.searchModuleDef() took 30252 milliseconds.
> After patch, XModuleDefService.searchModuleDef() took 13766 milliseconds.
>
> GET API /service/xusers/permission response improved by ~16 seconds for the above mentioned dataset.
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73452: RANGER-3023: Permission tab takes longer
time to load with large number of users and group_users data
Posted by Sailaja Polavarapu <sp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73452/#review223261
-----------------------------------------------------------
As an extension, since only username & groupname are used from XXUser & XXGroup objects, it is possible to add sql query to return map of id & name instead of the actual objects?
- Sailaja Polavarapu
On July 20, 2021, 1:52 p.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73452/
> -----------------------------------------------------------
>
> (Updated July 20, 2021, 1:52 p.m.)
>
>
> Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3023
> https://issues.apache.org/jira/browse/RANGER-3023
>
>
> Repository: ranger
>
>
> Description
> -------
>
> GET API /service/xusers/permission takes longer time to load with following number of users and group mappings in db.
>
> select count(*) from x_user;
> 109040
>
> select count(*) from x_portal_user;
> 109038
>
> select count(*) from x_group_users;
> 689952
>
> Current problem : For every ModuleDef, db call to fetch all XXUser, XXPortalUser and creating a Map object using xUserService.getXXPortalUserIdXXUserMap() is a costly operation. Similarly for xGroupService.getXXGroupIdXXGroupMap().
>
> Solution:
> In the following patch, I have overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once.
> i.e. Map<Long, XXUser> xXPortalUserIdXXUserMap = xUserService.getXXPortalUserIdXXUserMap();
> Map<Long, XXGroup> xXGroupMap = xGroupService.getXXGroupIdXXGroupMap();
>
> These two objects will be passed to an overloaded method populateViewBean()
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java d5ca38548
>
>
> Diff: https://reviews.apache.org/r/73452/diff/2/
>
>
> Testing
> -------
>
> xUserService.getXXPortalUserIdXXUserMap() takes approximately 2000 milliseconds.
> xGroupService.getXXGroupIdXXGroupMap() takes approximately 500 milliseconds.
>
> Before patch, XModuleDefServiceBase.searchModuleDef() took 30252 milliseconds.
> After patch, XModuleDefService.searchModuleDef() took 13766 milliseconds.
>
> GET API /service/xusers/permission response improved by ~16 seconds for the above mentioned dataset.
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73452: RANGER-3023: Permission tab takes longer
time to load with large number of users and group_users data
Posted by Mehul Parikh <xs...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73452/#review223694
-----------------------------------------------------------
Ship it!
Ship It!
- Mehul Parikh
On Oct. 26, 2021, 11:29 a.m., Mahesh Bandal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73452/
> -----------------------------------------------------------
>
> (Updated Oct. 26, 2021, 11:29 a.m.)
>
>
> Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3023
> https://issues.apache.org/jira/browse/RANGER-3023
>
>
> Repository: ranger
>
>
> Description
> -------
>
> GET API /service/xusers/permission takes longer time to load with following number of users and group mappings in db.
>
> select count(*) from x_user;
> 109040
>
> select count(*) from x_portal_user;
> 109038
>
> select count(*) from x_group_users;
> 689952
>
> Current problem : For every ModuleDef, db call to fetch all XXUser, XXPortalUser and creating a Map object using xUserService.getXXPortalUserIdXXUserMap() is a costly operation. Similarly for xGroupService.getXXGroupIdXXGroupMap().
>
> Solution:
> In the following patch, I have overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once.
> i.e. Map<Long, XXUser> xXPortalUserIdXXUserMap = xUserService.getXXPortalUserIdXXUserMap();
> Map<Long, XXGroup> xXGroupMap = xGroupService.getXXGroupIdXXGroupMap();
>
> These two objects will be passed to an overloaded method populateViewBean()
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2
> security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 4c0f33ed9
> security-admin/src/main/java/org/apache/ranger/service/XGroupPermissionService.java 13d0a8fcb
> security-admin/src/main/java/org/apache/ranger/service/XGroupService.java d615d1775
> security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java d5ca38548
> security-admin/src/main/java/org/apache/ranger/service/XUserPermissionService.java 47a1fadb7
> security-admin/src/main/java/org/apache/ranger/service/XUserService.java 9647096fe
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 96dc0df8e
>
>
> Diff: https://reviews.apache.org/r/73452/diff/3/
>
>
> Testing
> -------
>
> xUserService.getXXPortalUserIdXXUserMap() takes approximately 2000 milliseconds.
> xGroupService.getXXGroupIdXXGroupMap() takes approximately 500 milliseconds.
>
> Before patch, XModuleDefServiceBase.searchModuleDef() took 30252 milliseconds.
> After patch, XModuleDefService.searchModuleDef() took 13766 milliseconds.
>
> GET API /service/xusers/permission response improved by ~16 seconds for the above mentioned dataset.
>
>
> Thanks,
>
> Mahesh Bandal
>
>
Re: Review Request 73452: RANGER-3023: Permission tab takes longer
time to load with large number of users and group_users data
Posted by Mahesh Bandal <ma...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73452/
-----------------------------------------------------------
(Updated Oct. 26, 2021, 11:29 a.m.)
Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressing review comments.
Bugs: RANGER-3023
https://issues.apache.org/jira/browse/RANGER-3023
Repository: ranger
Description
-------
GET API /service/xusers/permission takes longer time to load with following number of users and group mappings in db.
select count(*) from x_user;
109040
select count(*) from x_portal_user;
109038
select count(*) from x_group_users;
689952
Current problem : For every ModuleDef, db call to fetch all XXUser, XXPortalUser and creating a Map object using xUserService.getXXPortalUserIdXXUserMap() is a costly operation. Similarly for xGroupService.getXXGroupIdXXGroupMap().
Solution:
In the following patch, I have overriedden searchModuleDef function in XModuleDefService which will fetch users and groups only once.
i.e. Map<Long, XXUser> xXPortalUserIdXXUserMap = xUserService.getXXPortalUserIdXXUserMap();
Map<Long, XXGroup> xXGroupMap = xGroupService.getXXGroupIdXXGroupMap();
These two objects will be passed to an overloaded method populateViewBean()
Diffs (updated)
-----
security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 1bd59f8d2
security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 4c0f33ed9
security-admin/src/main/java/org/apache/ranger/service/XGroupPermissionService.java 13d0a8fcb
security-admin/src/main/java/org/apache/ranger/service/XGroupService.java d615d1775
security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java d5ca38548
security-admin/src/main/java/org/apache/ranger/service/XUserPermissionService.java 47a1fadb7
security-admin/src/main/java/org/apache/ranger/service/XUserService.java 9647096fe
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 96dc0df8e
Diff: https://reviews.apache.org/r/73452/diff/3/
Changes: https://reviews.apache.org/r/73452/diff/2-3/
Testing
-------
xUserService.getXXPortalUserIdXXUserMap() takes approximately 2000 milliseconds.
xGroupService.getXXGroupIdXXGroupMap() takes approximately 500 milliseconds.
Before patch, XModuleDefServiceBase.searchModuleDef() took 30252 milliseconds.
After patch, XModuleDefService.searchModuleDef() took 13766 milliseconds.
GET API /service/xusers/permission response improved by ~16 seconds for the above mentioned dataset.
Thanks,
Mahesh Bandal