You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2019/09/25 18:26:12 UTC

Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/
-----------------------------------------------------------

Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Bugs: RANGER-2510
    https://issues.apache.org/jira/browse/RANGER-2510


Repository: ranger


Description
-------

Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
1. Compute time for rebuilding
2. Large traffic from ranger-admin to each of the plugins
3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.

It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.


Diffs
-----

  agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java 01ee945f3 
  agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b596992aa 
  agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java 74e7addbe 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java d33f5d36b 
  agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java aec325c03 
  agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java 633ec96ac 
  agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java 8de0329f6 
  agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java fe4b27817 
  agents-common/src/main/java/org/apache/ranger/plugin/store/TagValidator.java c334e3963 
  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java 310f69dbf 
  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java 605a74ae4 
  agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java PRE-CREATION 
  agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java fed3f1218 
  agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 7180675af 
  agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngineComparison.java PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/fail/myServicePolicies.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/fail/myServiceTags.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/fail/otherServicePolicies.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/fail/otherServiceTags.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/success/myServicePolicies.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/success/myServiceTags.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/success/otherServicePolicies.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/comparison/success/otherServiceTags.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/test_compare_policyengines.json PRE-CREATION 
  agents-common/src/test/resources/policyengine/test_policyengine_hive.json 66b3644b9 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json fce157262 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json e60bfe9b7 
  agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json dea1c2b74 
  knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java db16d73a0 
  security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 399fd8aaf 
  security-admin/db/mysql/patches/043-add-tag-change-log-table.sql PRE-CREATION 
  security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 73b91736c 
  security-admin/db/oracle/patches/043-add-tag-change-log-table.sql PRE-CREATION 
  security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 0e8fd9c13 
  security-admin/db/postgres/patches/043-add-tag-change-log-table.sql PRE-CREATION 
  security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 5d37f08d5 
  security-admin/db/sqlanywhere/patches/043-add-tag-change-log-table.sql PRE-CREATION 
  security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 16600e010 
  security-admin/db/sqlserver/patches/043-add-tag-change-log-table.sql PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 394c05ccf 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1c45788b 
  security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3cc476532 
  security-admin/src/main/java/org/apache/ranger/common/RangerAdminTagEnricher.java f81184d13 
  security-admin/src/main/java/org/apache/ranger/common/RangerServiceTagsCache.java 802f42a25 
  security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java e4e335fe1 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java ef436a0d7 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 808170323 
  security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 8ee181afb 
  security-admin/src/main/java/org/apache/ranger/rest/TagREST.java 03d06e248 
  security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java f0cb8f4d2 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java c04c3351a 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagResourceMapService.java 89c451ee2 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java f5060e532 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 078588a1b 
  security-admin/src/test/java/org/apache/ranger/rest/TestTagREST.java 90060b2a6 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 5032c12e5 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagResourceMapService.java 27ec8e140 


Diff: https://reviews.apache.org/r/71542/diff/1/


Testing
-------

Developed unit tests to verify that incremental policy and tag work as expected.


Thanks,

Abhay Kulkarni


Re: Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

Posted by Sailaja Polavarapu <sp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/#review217943
-----------------------------------------------------------


Ship it!




Ship It!

- Sailaja Polavarapu


On Sept. 25, 2019, 6:26 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71542/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2019, 6:26 p.m.)
> 
> 
> Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2510
>     https://issues.apache.org/jira/browse/RANGER-2510
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.
> 
> It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java 01ee945f3 
>   agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b596992aa 
>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java 74e7addbe 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java d33f5d36b 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java aec325c03 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java 633ec96ac 
>   agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java 8de0329f6 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java fe4b27817 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagValidator.java c334e3963 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java 310f69dbf 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java 605a74ae4 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java PRE-CREATION 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java fed3f1218 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 7180675af 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngineComparison.java PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_compare_policyengines.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive.json 66b3644b9 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json fce157262 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json e60bfe9b7 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json dea1c2b74 
>   knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java db16d73a0 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 399fd8aaf 
>   security-admin/db/mysql/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 73b91736c 
>   security-admin/db/oracle/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 0e8fd9c13 
>   security-admin/db/postgres/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 5d37f08d5 
>   security-admin/db/sqlanywhere/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 16600e010 
>   security-admin/db/sqlserver/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 394c05ccf 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1c45788b 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3cc476532 
>   security-admin/src/main/java/org/apache/ranger/common/RangerAdminTagEnricher.java f81184d13 
>   security-admin/src/main/java/org/apache/ranger/common/RangerServiceTagsCache.java 802f42a25 
>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java e4e335fe1 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java ef436a0d7 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 808170323 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 8ee181afb 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java 03d06e248 
>   security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java f0cb8f4d2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java c04c3351a 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagResourceMapService.java 89c451ee2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java f5060e532 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 078588a1b 
>   security-admin/src/test/java/org/apache/ranger/rest/TestTagREST.java 90060b2a6 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 5032c12e5 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagResourceMapService.java 27ec8e140 
> 
> 
> Diff: https://reviews.apache.org/r/71542/diff/1/
> 
> 
> Testing
> -------
> 
> Developed unit tests to verify that incremental policy and tag work as expected.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Re: Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/#review217944
-----------------------------------------------------------


Ship it!




Ship It!

- Ramesh Mani


On Sept. 25, 2019, 6:26 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71542/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2019, 6:26 p.m.)
> 
> 
> Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2510
>     https://issues.apache.org/jira/browse/RANGER-2510
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.
> 
> It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java 01ee945f3 
>   agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b596992aa 
>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java 74e7addbe 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java d33f5d36b 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java aec325c03 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java 633ec96ac 
>   agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java 8de0329f6 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java fe4b27817 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagValidator.java c334e3963 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java 310f69dbf 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java 605a74ae4 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java PRE-CREATION 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java fed3f1218 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 7180675af 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngineComparison.java PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_compare_policyengines.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive.json 66b3644b9 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json fce157262 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json e60bfe9b7 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json dea1c2b74 
>   knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java db16d73a0 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 399fd8aaf 
>   security-admin/db/mysql/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 73b91736c 
>   security-admin/db/oracle/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 0e8fd9c13 
>   security-admin/db/postgres/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 5d37f08d5 
>   security-admin/db/sqlanywhere/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 16600e010 
>   security-admin/db/sqlserver/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 394c05ccf 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1c45788b 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3cc476532 
>   security-admin/src/main/java/org/apache/ranger/common/RangerAdminTagEnricher.java f81184d13 
>   security-admin/src/main/java/org/apache/ranger/common/RangerServiceTagsCache.java 802f42a25 
>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java e4e335fe1 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java ef436a0d7 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 808170323 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 8ee181afb 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java 03d06e248 
>   security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java f0cb8f4d2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java c04c3351a 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagResourceMapService.java 89c451ee2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java f5060e532 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 078588a1b 
>   security-admin/src/test/java/org/apache/ranger/rest/TestTagREST.java 90060b2a6 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 5032c12e5 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagResourceMapService.java 27ec8e140 
> 
> 
> Diff: https://reviews.apache.org/r/71542/diff/1/
> 
> 
> Testing
> -------
> 
> Developed unit tests to verify that incremental policy and tag work as expected.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Re: Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/#review217978
-----------------------------------------------------------




agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
Line 253 (original), 256 (patched)
<https://reviews.apache.org/r/71542/#comment305399>

    setServiceTags() method is about 200 lines long:-(. To make it easier to read/review, consider breaking this into multiple methods.



agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
Lines 502 (patched)
<https://reviews.apache.org/r/71542/#comment305400>

    Can 'enrichedServiceTags' be null? Refer to line #467.



security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java
Lines 60 (patched)
<https://reviews.apache.org/r/71542/#comment305412>

    "some record" or "first record"? The while loop below seems to expect the first record to match the given version - line #72. Please review and update.



security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java
Lines 67 (patched)
<https://reviews.apache.org/r/71542/#comment305411>

    For better readability, consider replacing "2" with a constant. Please review similar usage in #127 - #130; and other such places.



security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java
Line 76 (original), 76 (patched)
<https://reviews.apache.org/r/71542/#comment305410>

    if 'resourceId' and 'tagId' are not expected to be null, consider adding a warn() log to help in troubleshooting. Please review other methods for similr code.



security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java
Lines 134 (patched)
<https://reviews.apache.org/r/71542/#comment305409>

    Seems like an incomplete/unnecessary method. Please review and update/remove.



security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java
Line 129 (original), 146 (patched)
<https://reviews.apache.org/r/71542/#comment305408>

    lines #146 - #155 can be moved outside the 'for' loop, as value of versionType and tagChangeType remain the same across iterations. Please review and update.



security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java
Lines 55 (patched)
<https://reviews.apache.org/r/71542/#comment305407>

    "some record" or "first record"? The while loop below seems to expect the first record to match the given version - line #67. Please review and update.



security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java
Lines 62 (patched)
<https://reviews.apache.org/r/71542/#comment305406>

    For better readability, consider replacing "2" with a constant. Please review similar usage in #106 - #110; and other such places.



security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java
Lines 183 (patched)
<https://reviews.apache.org/r/71542/#comment305404>

    Consider using Objects.equals(), for a simpler comparision code:
    
      return Objects.equals(this.id, other.id) &&
             Objects.equals(this.createTime, other.createTime) &&
             Objects.equals(this.serviceId, other.serviceId) &&
             ...



security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java
Lines 207 (patched)
<https://reviews.apache.org/r/71542/#comment305405>

    Consider replacing calls to this method with Objects.equals(object1, object2), and remove this method.



security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java
Line 529 (original), 529 (patched)
<https://reviews.apache.org/r/71542/#comment305403>

    Similar to deletePolicyDeltas(), a method to deleteTagDeltas() would need to be exposed. Please review and update.



security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java
Line 530 (original), 530 (patched)
<https://reviews.apache.org/r/71542/#comment305402>

    reloadServicePoliciesCache - this parameter is no more used. Please review and remove.



security-admin/src/main/resources/META-INF/jpa_named_queries.xml
Lines 1566 (patched)
<https://reviews.apache.org/r/71542/#comment305401>

    XXTagChangeLog.findByServiceId - this query doesn't seem to be used. Please review and remove if unused.


- Madhan Neethiraj


On Sept. 25, 2019, 6:26 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71542/
> -----------------------------------------------------------
> 
> (Updated Sept. 25, 2019, 6:26 p.m.)
> 
> 
> Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2510
>     https://issues.apache.org/jira/browse/RANGER-2510
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.
> 
> It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java 01ee945f3 
>   agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b596992aa 
>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerPolicyDelta.java 74e7addbe 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java d33f5d36b 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java aec325c03 
>   agents-common/src/main/java/org/apache/ranger/plugin/policyresourcematcher/RangerDefaultPolicyResourceMatcher.java 633ec96ac 
>   agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java 8de0329f6 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java fe4b27817 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagValidator.java c334e3963 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerRESTUtils.java 310f69dbf 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerResourceTrie.java 605a74ae4 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java PRE-CREATION 
>   agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java fed3f1218 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 7180675af 
>   agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngineComparison.java PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/fail/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/myServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServicePolicies.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/comparison/success/otherServiceTags.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_compare_policyengines.json PRE-CREATION 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive.json 66b3644b9 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_add.json fce157262 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_delete.json e60bfe9b7 
>   agents-common/src/test/resources/policyengine/test_policyengine_hive_incremental_update.json dea1c2b74 
>   knox-agent/src/main/java/org/apache/ranger/admin/client/RangerAdminJersey2RESTClient.java db16d73a0 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 399fd8aaf 
>   security-admin/db/mysql/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 73b91736c 
>   security-admin/db/oracle/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 0e8fd9c13 
>   security-admin/db/postgres/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 5d37f08d5 
>   security-admin/db/sqlanywhere/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 16600e010 
>   security-admin/db/sqlserver/patches/043-add-tag-change-log-table.sql PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 394c05ccf 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java e1c45788b 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3cc476532 
>   security-admin/src/main/java/org/apache/ranger/common/RangerAdminTagEnricher.java f81184d13 
>   security-admin/src/main/java/org/apache/ranger/common/RangerServiceTagsCache.java 802f42a25 
>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java e4e335fe1 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java ef436a0d7 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 808170323 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 2a4c53b48 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 8ee181afb 
>   security-admin/src/main/java/org/apache/ranger/rest/TagREST.java 03d06e248 
>   security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java f0cb8f4d2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java c04c3351a 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagResourceMapService.java 89c451ee2 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java f5060e532 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 078588a1b 
>   security-admin/src/test/java/org/apache/ranger/rest/TestTagREST.java 90060b2a6 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 5032c12e5 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagResourceMapService.java 27ec8e140 
> 
> 
> Diff: https://reviews.apache.org/r/71542/diff/1/
> 
> 
> Testing
> -------
> 
> Developed unit tests to verify that incremental policy and tag work as expected.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Re: Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/#review217998
-----------------------------------------------------------


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 30, 2019, 10 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71542/
> -----------------------------------------------------------
> 
> (Updated Sept. 30, 2019, 10 p.m.)
> 
> 
> Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2510
>     https://issues.apache.org/jira/browse/RANGER-2510
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.
> 
> It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java 306388534 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java 7055391d1 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 9f6f024bb 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java ef65e9a6b 
>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyChangeLog.java df87f70a5 
>   security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java c3eb14414 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java c33841dfd 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java d4350b68e 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 9714fa91b 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 160a53c50 
> 
> 
> Diff: https://reviews.apache.org/r/71542/diff/2/
> 
> 
> Testing
> -------
> 
> Developed unit tests to verify that incremental policy and tag work as expected.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Re: Review Request 71542: RANGER-2510: Support for Incremental tag updates to improve performance

Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71542/
-----------------------------------------------------------

(Updated Sept. 30, 2019, 10 p.m.)


Review request for ranger, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
-------

Addressed review comments


Bugs: RANGER-2510
    https://issues.apache.org/jira/browse/RANGER-2510


Repository: ranger


Description
-------

Currently, every change to any tag/service-resource/service-resource->tag mapping causes complete rebuilding of portions of policy-engine that map accessed resource to their tags. There are several disadvantages:
1. Compute time for rebuilding
2. Large traffic from ranger-admin to each of the plugins
3. Large load on JVM memory system because of frequent complete rebuilding of portions of policy-engine.

It will be more optimal to communicate only the changes to tags and apply them to existing policy-engine.


Diffs (updated)
-----

  agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java 306388534 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyChangeLogDao.java 7055391d1 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 9f6f024bb 
  security-admin/src/main/java/org/apache/ranger/db/XXTagChangeLogDao.java ef65e9a6b 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyChangeLog.java df87f70a5 
  security-admin/src/main/java/org/apache/ranger/entity/XXTagChangeLog.java c3eb14414 
  security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java c33841dfd 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java d4350b68e 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 9714fa91b 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 160a53c50 


Diff: https://reviews.apache.org/r/71542/diff/2/

Changes: https://reviews.apache.org/r/71542/diff/1-2/


Testing
-------

Developed unit tests to verify that incremental policy and tag work as expected.


Thanks,

Abhay Kulkarni