You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@freestoneinfotech.com> on 2016/12/13 09:28:24 UTC

Review Request 54619: RANGER-1237 : Ranger permissions page do not load when there are too many users

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54619/
-----------------------------------------------------------

Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.


Bugs: RANGER-1237
    https://issues.apache.org/jira/browse/RANGER-1237


Repository: ranger


Description
-------

**Problem Statement :** Ranger permissions page do not load when there are too many users.
Module permission edit page's select groups and users drop-down taking too much time to load and sometimes browser hangs when it contains a lot of users/groups.
When users/groups > 2000 it's talking almost 5-6 min to load users/groups in the dropdown. It is happening because for every user permission entry there is a db call which get details of users/groups to populate view bean from XXUserPermission/XXGroupPermission entity bean.

** Proposed Solution:** Rather fetching each individual user details in an individual db call we can collect all users details through one db call in a java map where key could be user or group id and value could be XXUser/XXGroup object. We can refer these map during bean conversion and load the user/group details in entity/view bean whenever required.

** Scope of improvement :** Currently Proposed solution is implemented in XModuleDefService.populateViewBean(XXModuleDef xObj) method where we load all user/group details map by making a db call. populateViewBean method can process user/group permissions of one module only at a time, to process all modules permissions XModuleDefServiceBase.searchModuleDef() method is having for loop for all the module and in that loop it makes call to populateViewBean() method. since populateViewBean() method is loading user/groups details on each call it might be doing the same for each module. currently there are 6 modules so we are loading same user/groups details 6 times to process all the modules. if we could pass user/group details map from  populateViewBean() then we dont have to load the user/group details map inside of populateViewBean() and it can be done in XModuleDefServiceBase.searchModuleDef() itself. This approach shall reduce 6 db calls to 1.

Please note that XModuleDefService.populateViewBean(XXModuleDef xObj) is being called from XUserMgr.updateXModuleDefPermission() method also.


Diffs
-----

  security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java e3f7223 
  security-admin/src/main/java/org/apache/ranger/service/XGroupPermissionService.java 20c3b67 
  security-admin/src/main/java/org/apache/ranger/service/XGroupService.java d74d2bf 
  security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java 2e00643 
  security-admin/src/main/java/org/apache/ranger/service/XUserPermissionService.java bd3a50d 
  security-admin/src/main/java/org/apache/ranger/service/XUserService.java 8ba9ef6 
  security-admin/src/main/webapp/scripts/controllers/Controller.js fc56496 
  security-admin/src/main/webapp/scripts/models/XABaseModel.js 03b4e13 
  security-admin/src/main/webapp/scripts/modules/globalize/message/en.js 99a5424 
  security-admin/src/main/webapp/scripts/utils/XAUtils.js 14ac70b 
  security-admin/src/main/webapp/scripts/views/permissions/ModulePermissionCreate.js 02b879d 
  security-admin/src/main/webapp/scripts/views/permissions/ModulePermissionForm.js 1756955 
  security-admin/src/main/webapp/styles/xa.css 264e0ea 
  security-admin/src/main/webapp/templates/permissions/ModulePermissionForm_tmpl.html 1bf096b 

Diff: https://reviews.apache.org/r/54619/diff/


Testing
-------

**Steps Performed(Without patch):**
1. Installed and Started Ranger without patch and created 10000 users and 2000 groups and provided permission on two module; which shall make 20000 entries in x_user_module_perm table and 4000 entries in x_group_module_perm table.
2. Logged into Ranger Admin and clicked on 'Permission page'.
3. Permission page loading took approx 5 minute to process the result at server end and few more minutes to render the result on Ranger UI.
4. Retried previous step three more times and processing time in each attempt was approx 4-5 minute.

**Steps Performed(With patch):**
1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
2. After Starting Ranger; logged into Ranger Admin and clicked on 'Permission page'.

**Expected Behaviour:** Report page processing at server end should take less than 5 minutes at server end.

**Actual Behavior:** Report page processing at server end took only 13 seconds to render the result on Ranger UI.

Note : Retried the same three more times and processing time at server end in each attempt was ~13 seconds.

**Other Test Results :**

**1. Total Users=10000, Total Groups=2000**
Entries in x_user_module_perm table: 20000
Entries in x_group_module_perm table: 4000
Time Taken to load the permission page of all module(in seconds):13
Time Taken to save the permission page of one module(in seconds):15

**2. Total Users=20000, Total Groups=4000**
Entries in x_user_module_perm table: 40000
Entries in x_group_module_perm table: 8000
Time Taken to load the permission page of all module(in seconds):23
Time Taken to save the permission page of one module(in seconds):30

**3. Total Users=30000, Total Groups=6000**
Entries in x_user_module_perm table: 60000
Entries in x_group_module_perm table: 12000
Time Taken to load the permission page of all module(in seconds):33
Time Taken to save the permission page of one module(in seconds):45

**4. Total Users=40000, Total Groups=8000**
Entries in x_user_module_perm table: 80000
Entries in x_group_module_perm table: 16000
Time Taken to load the permission page of all module(in seconds):37
Time Taken to save the permission page of one module(in seconds):78

**5. Total Users=50000, Total Groups=10000**
Entries in x_user_module_perm table: 100000
Entries in x_group_module_perm table: 20000
Time Taken to load the permission page of all module(in seconds):47
Time Taken to save the permission page of one module(in seconds):96


Thanks,

Pradeep Agrawal


Re: Review Request 54619: RANGER-1237 : Ranger permissions page do not load when there are too many users

Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54619/#review159114
-----------------------------------------------------------


Ship it!




Ship It!

- Velmurugan Periasamy


On Dec. 13, 2016, 9:28 a.m., Pradeep Agrawal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54619/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2016, 9:28 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-1237
>     https://issues.apache.org/jira/browse/RANGER-1237
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> **Problem Statement :** Ranger permissions page do not load when there are too many users.
> Module permission edit page's select groups and users drop-down taking too much time to load and sometimes browser hangs when it contains a lot of users/groups.
> When users/groups > 2000 it's talking almost 5-6 min to load users/groups in the dropdown. It is happening because for every user permission entry there is a db call which get details of users/groups to populate view bean from XXUserPermission/XXGroupPermission entity bean.
> 
> ** Proposed Solution:** Rather fetching each individual user details in an individual db call we can collect all users details through one db call in a java map where key could be user or group id and value could be XXUser/XXGroup object. We can refer these map during bean conversion and load the user/group details in entity/view bean whenever required.
> 
> ** Scope of improvement :** Currently Proposed solution is implemented in XModuleDefService.populateViewBean(XXModuleDef xObj) method where we load all user/group details map by making a db call. populateViewBean method can process user/group permissions of one module only at a time, to process all modules permissions XModuleDefServiceBase.searchModuleDef() method is having for loop for all the module and in that loop it makes call to populateViewBean() method. since populateViewBean() method is loading user/groups details on each call it might be doing the same for each module. currently there are 6 modules so we are loading same user/groups details 6 times to process all the modules. if we could pass user/group details map from  populateViewBean() then we dont have to load the user/group details map inside of populateViewBean() and it can be done in XModuleDefServiceBase.searchModuleDef() itself. This approach shall reduce 6 db calls to 1.
> 
> Please note that XModuleDefService.populateViewBean(XXModuleDef xObj) is being called from XUserMgr.updateXModuleDefPermission() method also.
> 
> 
> Diffs
> -----
> 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java e3f7223 
>   security-admin/src/main/java/org/apache/ranger/service/XGroupPermissionService.java 20c3b67 
>   security-admin/src/main/java/org/apache/ranger/service/XGroupService.java d74d2bf 
>   security-admin/src/main/java/org/apache/ranger/service/XModuleDefService.java 2e00643 
>   security-admin/src/main/java/org/apache/ranger/service/XUserPermissionService.java bd3a50d 
>   security-admin/src/main/java/org/apache/ranger/service/XUserService.java 8ba9ef6 
>   security-admin/src/main/webapp/scripts/controllers/Controller.js fc56496 
>   security-admin/src/main/webapp/scripts/models/XABaseModel.js 03b4e13 
>   security-admin/src/main/webapp/scripts/modules/globalize/message/en.js 99a5424 
>   security-admin/src/main/webapp/scripts/utils/XAUtils.js 14ac70b 
>   security-admin/src/main/webapp/scripts/views/permissions/ModulePermissionCreate.js 02b879d 
>   security-admin/src/main/webapp/scripts/views/permissions/ModulePermissionForm.js 1756955 
>   security-admin/src/main/webapp/styles/xa.css 264e0ea 
>   security-admin/src/main/webapp/templates/permissions/ModulePermissionForm_tmpl.html 1bf096b 
> 
> Diff: https://reviews.apache.org/r/54619/diff/
> 
> 
> Testing
> -------
> 
> **Steps Performed(Without patch):**
> 1. Installed and Started Ranger without patch and created 10000 users and 2000 groups and provided permission on two module; which shall make 20000 entries in x_user_module_perm table and 4000 entries in x_group_module_perm table.
> 2. Logged into Ranger Admin and clicked on 'Permission page'.
> 3. Permission page loading took approx 5 minute to process the result at server end and few more minutes to render the result on Ranger UI.
> 4. Retried previous step three more times and processing time in each attempt was approx 4-5 minute.
> 
> **Steps Performed(With patch):**
> 1. Installed Ranger with patch; Provided same Ranger DB and user credentials to start the Ranger.
> 2. After Starting Ranger; logged into Ranger Admin and clicked on 'Permission page'.
> 
> **Expected Behaviour:** Report page processing at server end should take less than 5 minutes at server end.
> 
> **Actual Behavior:** Report page processing at server end took only 13 seconds to render the result on Ranger UI.
> 
> Note : Retried the same three more times and processing time at server end in each attempt was ~13 seconds.
> 
> **Other Test Results :**
> 
> **1. Total Users=10000, Total Groups=2000**
> Entries in x_user_module_perm table: 20000
> Entries in x_group_module_perm table: 4000
> Time Taken to load the permission page of all module(in seconds):13
> Time Taken to save the permission page of one module(in seconds):15
> 
> **2. Total Users=20000, Total Groups=4000**
> Entries in x_user_module_perm table: 40000
> Entries in x_group_module_perm table: 8000
> Time Taken to load the permission page of all module(in seconds):23
> Time Taken to save the permission page of one module(in seconds):30
> 
> **3. Total Users=30000, Total Groups=6000**
> Entries in x_user_module_perm table: 60000
> Entries in x_group_module_perm table: 12000
> Time Taken to load the permission page of all module(in seconds):33
> Time Taken to save the permission page of one module(in seconds):45
> 
> **4. Total Users=40000, Total Groups=8000**
> Entries in x_user_module_perm table: 80000
> Entries in x_group_module_perm table: 16000
> Time Taken to load the permission page of all module(in seconds):37
> Time Taken to save the permission page of one module(in seconds):78
> 
> **5. Total Users=50000, Total Groups=10000**
> Entries in x_user_module_perm table: 100000
> Entries in x_group_module_perm table: 20000
> Time Taken to load the permission page of all module(in seconds):47
> Time Taken to save the permission page of one module(in seconds):96
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>