You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Andrew Luo <an...@outlook.com> on 2020/02/26 22:14:22 UTC
Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/
-----------------------------------------------------------
Review request for ranger and Pradeep Agrawal.
Bugs: RANGER-2713
https://issues.apache.org/jira/browse/RANGER-2713
Repository: ranger
Description
-------
XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
Diffs
-----
security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
Diff: https://reviews.apache.org/r/72166/diff/1/
Testing
-------
Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
Also tested user deletion on MySQL:
1. Created new user
2. Logged in as new user, created a policy
3. Logged back in as original user, delete user that created policy
4. Verified no errors, policy is still present/correct
Thanks,
Andrew Luo
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Andrew Luo <an...@outlook.com>.
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote:
> > From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
>
> Andrew Luo wrote:
> I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results.
>
> Andrew Luo wrote:
> Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar test:
>
> 1. Install 0.7.0
> 2. Created new user
> 3. Logged in as new user, created a policy
> 4. Upgraded to my local build
> 5. Logged back in as original user, delete user that created policy
> 6. Verified no errors, policy is still present/correct
>
> I found some other bugs during the upgrade with the patches related to Kafka (J10015, J10025, J10033) but the error was showing it was failing to insert into x_access_type_def. I didn't change that table in my patch so I think its a different issue (I will file a bug later if I repro this on a build without my changes). Anyways, during the upgrade I just skipped those patches by manually inserting those records into the table so the upgrade script would think those patches already got applied, and everything worked as expected.
(I also tested policy creation after upgrade)
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Andrew Luo <an...@outlook.com>.
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote:
> > From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
>
> Andrew Luo wrote:
> I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results.
>
> Andrew Luo wrote:
> Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar test:
>
> 1. Install 0.7.0
> 2. Created new user
> 3. Logged in as new user, created a policy
> 4. Upgraded to my local build
> 5. Logged back in as original user, delete user that created policy
> 6. Verified no errors, policy is still present/correct
>
> I found some other bugs during the upgrade with the patches related to Kafka (J10015, J10025, J10033) but the error was showing it was failing to insert into x_access_type_def. I didn't change that table in my patch so I think its a different issue (I will file a bug later if I repro this on a build without my changes). Anyways, during the upgrade I just skipped those patches by manually inserting those records into the table so the upgrade script would think those patches already got applied, and everything worked as expected.
>
> Andrew Luo wrote:
> (I also tested policy creation after upgrade)
>
> Pradeep Agrawal wrote:
> Thanks Andrew. I will also see what are the issues while upgrading from ranger-0.7 to ranger-2.1
I found the issue: https://reviews.apache.org/r/72205
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Andrew Luo <an...@outlook.com>.
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote:
> > From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Andrew Luo <an...@outlook.com>.
> On March 3, 2020, 12:28 p.m., Pradeep Agrawal wrote:
> > From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
>
> Andrew Luo wrote:
> I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results.
Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar test:
1. Install 0.7.0
2. Created new user
3. Logged in as new user, created a policy
4. Upgraded to my local build
5. Logged back in as original user, delete user that created policy
6. Verified no errors, policy is still present/correct
I found some other bugs during the upgrade with the patches related to Kafka (J10015, J10025, J10033) but the error was showing it was failing to insert into x_access_type_def. I didn't change that table in my patch so I think its a different issue (I will file a bug later if I repro this on a build without my changes). Anyways, during the upgrade I just skipped those patches by manually inserting those records into the table so the upgrade script would think those patches already got applied, and everything worked as expected.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
On Feb. 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On 三月 3, 2020, 12:28 p.m., Pradeep Agrawal wrote:
> > From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
>
> Andrew Luo wrote:
> I tested from the previous build (without my change) to a version with my change. I'll also test from 0.7 and let you know the results.
>
> Andrew Luo wrote:
> Done, I tested from 0.7.0 to my local build (on MySQL) and did a similar test:
>
> 1. Install 0.7.0
> 2. Created new user
> 3. Logged in as new user, created a policy
> 4. Upgraded to my local build
> 5. Logged back in as original user, delete user that created policy
> 6. Verified no errors, policy is still present/correct
>
> I found some other bugs during the upgrade with the patches related to Kafka (J10015, J10025, J10033) but the error was showing it was failing to insert into x_access_type_def. I didn't change that table in my patch so I think its a different issue (I will file a bug later if I repro this on a build without my changes). Anyways, during the upgrade I just skipped those patches by manually inserting those records into the table so the upgrade script would think those patches already got applied, and everything worked as expected.
>
> Andrew Luo wrote:
> (I also tested policy creation after upgrade)
Thanks Andrew. I will also see what are the issues while upgrading from ranger-0.7 to ranger-2.1
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
On 二月 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated 二月 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/#review219731
-----------------------------------------------------------
From which version upgrade case has been tested ? can you try from ranger-0.7 also if it was not tested?
- Pradeep Agrawal
On 二月 26, 2020, 10:18 p.m., Andrew Luo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72166/
> -----------------------------------------------------------
>
> (Updated 二月 26, 2020, 10:18 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-2713
> https://issues.apache.org/jira/browse/RANGER-2713
>
>
> Repository: ranger
>
>
> Description
> -------
>
> XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
>
>
> Diffs
> -----
>
> security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
> security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
> security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
> security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
> security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
> security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
> security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
>
>
> Diff: https://reviews.apache.org/r/72166/diff/1/
>
>
> Testing
> -------
>
> Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
>
> Also tested user deletion on MySQL:
>
> 1. Created new user
> 2. Logged in as new user, created a policy
> 3. Logged back in as original user, delete user that created policy
> 4. Verified no errors, policy is still present/correct
>
> Also searched the source code and JPA queries xml for other references to these fields
>
>
> Thanks,
>
> Andrew Luo
>
>
Re: Review Request 72166: RANGER-2713: Remove audit fields from
XXPolicyRef objects
Posted by Andrew Luo <an...@outlook.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72166/
-----------------------------------------------------------
(Updated Feb. 26, 2020, 10:18 p.m.)
Review request for ranger and Pradeep Agrawal.
Bugs: RANGER-2713
https://issues.apache.org/jira/browse/RANGER-2713
Repository: ranger
Description
-------
XXPolicyRef objects have fields such as create time, update time, added by user ID, updated by user ID, but there fields are entirely useless since they are all copied from the XXPolicy object. In addition, while improving performance for creation of policies with large numbers of users, we discovered that a lot of time was being spent in JPA converting these Date objects especially. After removing these fields we saw a significant performance improvement (a secondary benefit is less database space usage).
Diffs
-----
security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 47618f6b3
security-admin/db/mysql/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql e59e7de61
security-admin/db/oracle/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 37ea61912
security-admin/db/postgres/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql bd1c47cc4
security-admin/db/sqlanywhere/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 22e1746f2
security-admin/db/sqlserver/patches/046-drop-audit-columns-from-policy-ref-tables.sql PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java baacfa4d8
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java 6af8f99f4
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java 4f4409d6a
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java cb926740e
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java 32a1b9f24
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java 115064621
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefRole.java 7aee502e0
security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java 8dfb92833
security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java 85e457efa
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 69c8a4cb6
Diff: https://reviews.apache.org/r/72166/diff/1/
Testing (updated)
-------
Fresh install and upgrade tested on all 5 databases (tested service and policy creation)
Also tested user deletion on MySQL:
1. Created new user
2. Logged in as new user, created a policy
3. Logged back in as original user, delete user that created policy
4. Verified no errors, policy is still present/correct
Also searched the source code and JPA queries xml for other references to these fields
Thanks,
Andrew Luo