You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "Ramachandran (Jira)" <ji...@apache.org> on 2023/02/09 03:56:00 UTC
[jira] [Updated] (RANGER-3899) Policy creation takes more time when there are more users,groups,roles
[ https://issues.apache.org/jira/browse/RANGER-3899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramachandran updated RANGER-3899:
---------------------------------
Description:
*{color:#0000ff}Policy Creation Steps in Apache Ranger:{color}*{color:#0000ff} {color}
{code:java}
1. Get the service by Name:
RangerService service = getServiceByName(policy.getService());→ 1DB Read call
2. Get XXServiceDef by Name
XXServiceDef xServiceDef = daoMgr.getXXServiceDef().findByName(service.getType()); → 1DB Read call
3. Get the existing XXPolicy by Name
XXPolicy existing = daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(), service.getId(), zoneId); → 1DB Read call
4. Create a policy
policy = policyService.create(policy, true); → 1DB Write call
5. Get policy by Id
XXPolicy xCreatedPolicy = daoMgr.getXXPolicy().getById(policy.getId()); → 1DB Read call
6. createObjectDataHistory for the newly created policy
dataHistService.createObjectDataHistory(createdPolicy, RangerDataHistService.ACTION_CREATE); → 1DB Write call
7. createTrxLog for the newly created policy
bizUtil.createTrxLog(trxLogList); → 1DB Write call
8. Create Policy Label for the newly created policy
createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); → 1DB Write call
9. Create Policy Mapping For Ref Table
policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy, xServiceDef);
1. Getting roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from each policy items of the policy
2. Getting resourceNames from the policy
3.For each resource
Getting the XXResourceDef by resource and policy.getId()
XXResourceDef xResDef = daoMgr.getXXResourceDef().findByNameAndPolicyId(resource, policy.getId()); → 1DB Read call for each resource
populate XXPolicyRefResource
4. Create the batch of XXPolicyRefResource in DB
daoMgr.getXXPolicyRefResource().batchCreate(xPolResources); → 1DB Batch Write call
5.For each roleNames
Getting XXRole by roleName
XXRole xRole = daoMgr.getXXRole().findByRoleName(role); → 1DB Read call for each role
populate XXPolicyRefRole
6. Create the batch of XXPolicyRefRole in DB
daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles); → 1DB Batch Write call
7.For each groupNames
Getting the XXGroup by groupName
XXGroup xGroup = daoMgr.getXXGroup().findByGroupName(context.group.getName());→ 1DB Read call for each group
populate XXPolicyRefGroup
Insert into DB
daoMgr.getXXPolicyRefGroup().create(xPolGroup);→ 1DB write call for each group
8.For each userNames
Getting XXUser by userName
XXUser xUser = daoMgr.getXXUser().findByUserName(user); → 1DB Read call for each user
populate XXPolicyRefUser
daoMgr.getXXPolicyRefUser().create(xPolUser); → 1DB write call for each user
9.For each accessTypes
Getting the XXAccessTypeDef by accessType and xPolicy.getService()
XXAccessTypeDef xAccTypeDef = daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType, xPolicy.getService()); → 1DB Read call for each accesType
populate XXAccessTypeDef
10. Create the batch of xPolAccesses in DB
daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses); → 1DB Batch Write call
11.For each conditionTypes
Getting the XXPolicyConditionDef by condition and xServiceDef.getId()
XXPolicyConditionDef xPolCondDef = daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(), condition); → 1DB Read call for each conditionType
populate XXPolicyConditionDef
12. Create the batch of xPolConds in DB
daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds); → 1DB Batch Write call
{code}
*{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(Before Fix):{color}*
The Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|504|526|3|{color:#ff0000}4~8 seconds{color}|
*{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(After Fix):{color}*
The policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
{color:#ff0000}Proposal :{color}
{color:#ff0000}After changing to batch create of XXPolicyRefUser instead of calling 1 DB write XXPolicyRefUser for every user of the policy:{color}
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|6|526|5|{color:#ff0000}2.6~4 seconds{color}|
A similar will be applied for roles and groups as well.
After fixing (RANGER-3899, RANGER-2732,)
When Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
After the Batch read the calls will be reduced into
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|6|27|5|<2.6 seconds |
DB read calls will be reduced to {color:#de350b}27{color} from {color:#403294}526{color}
{color:#403294}Fixing this part will reduce the DB read calls to a very minimum which will improve performance drastically.{color}
was:
*{color:#0000ff}Policy Creation Steps in Apache Ranger:{color}*{color:#0000ff} {color}
{code:java}
1. Get the service by Name:
RangerService service = getServiceByName(policy.getService());→ 1DB Read call
2. Get XXServiceDef by Name
XXServiceDef xServiceDef = daoMgr.getXXServiceDef().findByName(service.getType()); → 1DB Read call
3. Get the existing XXPolicy by Name
XXPolicy existing = daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(), service.getId(), zoneId); → 1DB Read call
4. Create a policy
policy = policyService.create(policy, true); → 1DB Write call
5. Get policy by Id
XXPolicy xCreatedPolicy = daoMgr.getXXPolicy().getById(policy.getId()); → 1DB Read call
6. createObjectDataHistory for the newly created policy
dataHistService.createObjectDataHistory(createdPolicy, RangerDataHistService.ACTION_CREATE); → 1DB Write call
7. createTrxLog for the newly created policy
bizUtil.createTrxLog(trxLogList); → 1DB Write call
8. Create Policy Label for the newly created policy
createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); → 1DB Write call
9. Create Policy Mapping For Ref Table
policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy, xServiceDef);
1. Getting roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from each policy items of the policy
2. Getting resourceNames from the policy
3.For each resource
Getting the XXResourceDef by resource and policy.getId()
XXResourceDef xResDef = daoMgr.getXXResourceDef().findByNameAndPolicyId(resource, policy.getId()); → 1DB Read call for each resource
populate XXPolicyRefResource
4. Create the batch of XXPolicyRefResource in DB
daoMgr.getXXPolicyRefResource().batchCreate(xPolResources); → 1DB Batch Write call
5.For each roleNames
Getting XXRole by roleName
XXRole xRole = daoMgr.getXXRole().findByRoleName(role); → 1DB Read call for each role
populate XXPolicyRefRole
6. Create the batch of XXPolicyRefRole in DB
daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles); → 1DB Batch Write call
7.For each groupNames
Getting the XXGroup by groupName
XXGroup xGroup = daoMgr.getXXGroup().findByGroupName(context.group.getName());→ 1DB Read call for each group
populate XXPolicyRefGroup
Insert into DB
daoMgr.getXXPolicyRefGroup().create(xPolGroup);→ 1DB write call for each group
8.For each userNames
Getting XXUser by userName
XXUser xUser = daoMgr.getXXUser().findByUserName(user); → 1DB Read call for each user
populate XXPolicyRefUser
daoMgr.getXXPolicyRefUser().create(xPolUser); → 1DB write call for each user
9.For each accessTypes
Getting the XXAccessTypeDef by accessType and xPolicy.getService()
XXAccessTypeDef xAccTypeDef = daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType, xPolicy.getService()); → 1DB Read call for each accesType
populate XXAccessTypeDef
10. Create the batch of xPolAccesses in DB
daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses); → 1DB Batch Write call
11.For each conditionTypes
Getting the XXPolicyConditionDef by condition and xServiceDef.getId()
XXPolicyConditionDef xPolCondDef = daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(), condition); → 1DB Read call for each conditionType
populate XXPolicyConditionDef
12. Create the batch of xPolConds in DB
daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds); → 1DB Batch Write call
{code}
*{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(Before Fix):{color}*
The Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|504|526|3|{color:#ff0000}4~8 seconds{color}|
*{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(After Fix):{color}*
The policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
{color:#ff0000}Proposal :{color}
{color:#ff0000}After changing to batch create of XXPolicyRefUser instead of calling 1 DB write XXPolicyRefUser for every user of the policy:{color}
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|6|526|5|{color:#ff0000}2.6~4 seconds{color}|
A similar will be applied for roles and groups as well.
After fixing (RANGER-3899, RANGER-2732,)
When Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
After the Batch read the calls will be reduced into
||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
|6|27|5|<2.6 seconds (needs to be validated)|
DB read calls will be reduced to {color:#de350b}27{color} from {color:#403294}526{color}
{color:#403294}Fixing this part will reduce the DB read calls to a very minimum which will improve performance drastically.{color}
> Policy creation takes more time when there are more users,groups,roles
> ----------------------------------------------------------------------
>
> Key: RANGER-3899
> URL: https://issues.apache.org/jira/browse/RANGER-3899
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
> Affects Versions: 3.0.0
> Reporter: Ramachandran
> Assignee: Ramachandran
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: 0001-RANGER-3899-Policy-creation-takes-more-time-when-the.patch
>
>
> *{color:#0000ff}Policy Creation Steps in Apache Ranger:{color}*{color:#0000ff} {color}
>
> {code:java}
> 1. Get the service by Name:
> RangerService service = getServiceByName(policy.getService());→ 1DB Read call
> 2. Get XXServiceDef by Name
> XXServiceDef xServiceDef = daoMgr.getXXServiceDef().findByName(service.getType()); → 1DB Read call
> 3. Get the existing XXPolicy by Name
> XXPolicy existing = daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(), service.getId(), zoneId); → 1DB Read call
> 4. Create a policy
> policy = policyService.create(policy, true); → 1DB Write call
> 5. Get policy by Id
> XXPolicy xCreatedPolicy = daoMgr.getXXPolicy().getById(policy.getId()); → 1DB Read call
> 6. createObjectDataHistory for the newly created policy
> dataHistService.createObjectDataHistory(createdPolicy, RangerDataHistService.ACTION_CREATE); → 1DB Write call
> 7. createTrxLog for the newly created policy
> bizUtil.createTrxLog(trxLogList); → 1DB Write call
> 8. Create Policy Label for the newly created policy
> createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); → 1DB Write call
> 9. Create Policy Mapping For Ref Table
> policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy, xServiceDef);
> 1. Getting roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from each policy items of the policy
> 2. Getting resourceNames from the policy
> 3.For each resource
> Getting the XXResourceDef by resource and policy.getId()
> XXResourceDef xResDef = daoMgr.getXXResourceDef().findByNameAndPolicyId(resource, policy.getId()); → 1DB Read call for each resource
> populate XXPolicyRefResource
> 4. Create the batch of XXPolicyRefResource in DB
> daoMgr.getXXPolicyRefResource().batchCreate(xPolResources); → 1DB Batch Write call
> 5.For each roleNames
> Getting XXRole by roleName
> XXRole xRole = daoMgr.getXXRole().findByRoleName(role); → 1DB Read call for each role
> populate XXPolicyRefRole
> 6. Create the batch of XXPolicyRefRole in DB
> daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles); → 1DB Batch Write call
> 7.For each groupNames
> Getting the XXGroup by groupName
> XXGroup xGroup = daoMgr.getXXGroup().findByGroupName(context.group.getName());→ 1DB Read call for each group
> populate XXPolicyRefGroup
> Insert into DB
> daoMgr.getXXPolicyRefGroup().create(xPolGroup);→ 1DB write call for each group
> 8.For each userNames
> Getting XXUser by userName
> XXUser xUser = daoMgr.getXXUser().findByUserName(user); → 1DB Read call for each user
> populate XXPolicyRefUser
> daoMgr.getXXPolicyRefUser().create(xPolUser); → 1DB write call for each user
> 9.For each accessTypes
> Getting the XXAccessTypeDef by accessType and xPolicy.getService()
> XXAccessTypeDef xAccTypeDef = daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType, xPolicy.getService()); → 1DB Read call for each accesType
> populate XXAccessTypeDef
> 10. Create the batch of xPolAccesses in DB
> daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses); → 1DB Batch Write call
> 11.For each conditionTypes
> Getting the XXPolicyConditionDef by condition and xServiceDef.getId()
> XXPolicyConditionDef xPolCondDef = daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(), condition); → 1DB Read call for each conditionType
> populate XXPolicyConditionDef
> 12. Create the batch of xPolConds in DB
> daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds); → 1DB Batch Write call
> {code}
>
> *{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(Before Fix):{color}*
>
> The Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
>
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
> |504|526|3|{color:#ff0000}4~8 seconds{color}|
> *{color:#0000ff}A total number of DB calls involved for the below Policy creation in Apache Ranger(After Fix):{color}*
>
> The policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
> {color:#ff0000}Proposal :{color}
> {color:#ff0000}After changing to batch create of XXPolicyRefUser instead of calling 1 DB write XXPolicyRefUser for every user of the policy:{color}
>
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
> |6|526|5|{color:#ff0000}2.6~4 seconds{color}|
> A similar will be applied for roles and groups as well.
>
> After fixing (RANGER-3899, RANGER-2732,)
> When Policy contains 500 users,5 access Types (permissions),12 resources (1DB,1Table,10 columns)
> After the Batch read the calls will be reduced into
> ||DB Write count||DB Read count||DB Batch Write count||Time is taken to create the policy||
> |6|27|5|<2.6 seconds |
> DB read calls will be reduced to {color:#de350b}27{color} from {color:#403294}526{color}
> {color:#403294}Fixing this part will reduce the DB read calls to a very minimum which will improve performance drastically.{color}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)