You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2015/12/01 22:48:11 UTC
Review Request 40839: RANGER-753 - Instrumented policy download and
policy evaluation for performance measurement.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40839/
-----------------------------------------------------------
Review request for ranger and Madhan Neethiraj.
Bugs: RANGER-753
https://issues.apache.org/jira/browse/RANGER-753
Repository: ranger
Description
-------
Instrumented policy download and policy evaluation for performance measurement.
Optimized implementation of tag download.
- Reduced number of database queries to fetch Tag-related objects. The number of database queries is now constant; it does not depend on the number of Tag-related objects in the ranger-admin.
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerAbstractConditionEvaluator.java 0bcb744
agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerConditionEvaluator.java 602b80e
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b5662bf
agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java ba2a5d7
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java 8adab7b
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 57b1b7d
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerAbstractPolicyItemEvaluator.java ffd1d79
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerCachedPolicyEvaluator.java d67777c
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java da15c00
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyItemEvaluator.java 39a0a5e
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerOptimizedPolicyEvaluator.java e81280f
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerPolicyItemEvaluator.java 95d2b4e
agents-common/src/main/java/org/apache/ranger/plugin/resourcematcher/RangerAbstractResourceMatcher.java 56c4cfb
agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractTagStore.java ed1b64d
agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java ed20c51
agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java 0729339
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java b59ae1f
agents-common/src/test/resources/log4j.properties cb409e8
agents-common/src/test/resources/log4j.xml PRE-CREATION
security-admin/.gitignore bf7dc37
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4c65178
security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 300ba8d
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 8e2baab
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 04942a7
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 5ba3b74
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java c8cb91d
security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java 28ddfde
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java d92fd41
security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java b1986be
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4719fc9
security-admin/src/test/resources/log4j.properties cb409e8
security-admin/src/test/resources/log4j.xml PRE-CREATION
Diff: https://reviews.apache.org/r/40839/diff/
Testing
-------
Tested using python scripts to upload and download a large (of the order of tens of thousands) number of TagDefs, Tags, ServiceResources and TagResourceMap. Download of 10,000 tag-objects takes about 4 seconds in Ranger-Admin.
Thanks,
Abhay Kulkarni
Re: Review Request 40839: RANGER-753 - Instrumented policy download
and policy evaluation for performance measurement.
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40839/#review108602
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Dec. 2, 2015, 3:01 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40839/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 3:01 a.m.)
>
>
> Review request for ranger and Madhan Neethiraj.
>
>
> Bugs: RANGER-753
> https://issues.apache.org/jira/browse/RANGER-753
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Instrumented policy download and policy evaluation for performance measurement.
> Optimized implementation of tag download.
> - Reduced number of database queries to fetch Tag-related objects. The number of database queries is now constant; it does not depend on the number of Tag-related objects in the ranger-admin.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerAbstractConditionEvaluator.java 0bcb744
> agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerConditionEvaluator.java 602b80e
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b5662bf
> agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java ba2a5d7
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java 8adab7b
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 57b1b7d
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerAbstractPolicyItemEvaluator.java ffd1d79
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerCachedPolicyEvaluator.java d67777c
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java da15c00
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyItemEvaluator.java 39a0a5e
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerOptimizedPolicyEvaluator.java e81280f
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerPolicyItemEvaluator.java 95d2b4e
> agents-common/src/main/java/org/apache/ranger/plugin/resourcematcher/RangerAbstractResourceMatcher.java 56c4cfb
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractTagStore.java ed1b64d
> agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java ed20c51
> agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java 0729339
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java b59ae1f
> agents-common/src/test/resources/log4j.properties cb409e8
> agents-common/src/test/resources/log4j.xml PRE-CREATION
> security-admin/.gitignore bf7dc37
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4c65178
> security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 300ba8d
> security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 8e2baab
> security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 04942a7
> security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 5ba3b74
> security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java c8cb91d
> security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java 28ddfde
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java d92fd41
> security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java b1986be
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4719fc9
> security-admin/src/test/resources/log4j.properties cb409e8
> security-admin/src/test/resources/log4j.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40839/diff/
>
>
> Testing
> -------
>
> Tested using python scripts to upload and download a large (of the order of tens of thousands) number of TagDefs, Tags, ServiceResources and TagResourceMap. Download of 10,000 tag-objects takes about 4 seconds in Ranger-Admin.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 40839: RANGER-753 - Instrumented policy download
and policy evaluation for performance measurement.
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40839/
-----------------------------------------------------------
(Updated Dec. 2, 2015, 3:01 a.m.)
Review request for ranger and Madhan Neethiraj.
Changes
-------
Addressed review comments
Bugs: RANGER-753
https://issues.apache.org/jira/browse/RANGER-753
Repository: ranger
Description
-------
Instrumented policy download and policy evaluation for performance measurement.
Optimized implementation of tag download.
- Reduced number of database queries to fetch Tag-related objects. The number of database queries is now constant; it does not depend on the number of Tag-related objects in the ranger-admin.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerAbstractConditionEvaluator.java 0bcb744
agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerConditionEvaluator.java 602b80e
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b5662bf
agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java ba2a5d7
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java 8adab7b
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 57b1b7d
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerAbstractPolicyItemEvaluator.java ffd1d79
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerCachedPolicyEvaluator.java d67777c
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java da15c00
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyItemEvaluator.java 39a0a5e
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerOptimizedPolicyEvaluator.java e81280f
agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerPolicyItemEvaluator.java 95d2b4e
agents-common/src/main/java/org/apache/ranger/plugin/resourcematcher/RangerAbstractResourceMatcher.java 56c4cfb
agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractTagStore.java ed1b64d
agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java ed20c51
agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java 0729339
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java b59ae1f
agents-common/src/test/resources/log4j.properties cb409e8
agents-common/src/test/resources/log4j.xml PRE-CREATION
security-admin/.gitignore bf7dc37
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4c65178
security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 300ba8d
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 8e2baab
security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 04942a7
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 5ba3b74
security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java c8cb91d
security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java 28ddfde
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java d92fd41
security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java b1986be
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4719fc9
security-admin/src/test/resources/log4j.properties cb409e8
security-admin/src/test/resources/log4j.xml PRE-CREATION
Diff: https://reviews.apache.org/r/40839/diff/
Testing
-------
Tested using python scripts to upload and download a large (of the order of tens of thousands) number of TagDefs, Tags, ServiceResources and TagResourceMap. Download of 10,000 tag-objects takes about 4 seconds in Ranger-Admin.
Thanks,
Abhay Kulkarni
Re: Review Request 40839: RANGER-753 - Instrumented policy download
and policy evaluation for performance measurement.
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40839/#review108594
-----------------------------------------------------------
Ship it!
Please review the comments and update as necessary.
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java (line 57)
<https://reviews.apache.org/r/40839/#comment168048>
lastKnownVersion is already present in serviceTags. Please review if lastKnownVersion member can be removed from here.
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java (line 210)
<https://reviews.apache.org/r/40839/#comment168049>
hmm..this is not really a "copy". I think it will be enough to simply use a local variable with the same name:
final List<RangerServiceResourceMatcher> serviceResourceMatchers = this.serviceResourceMatchers;
final ServiceTags serviceTags = this.serviceTags;
agents-common/src/test/resources/log4j.xml (line 26)
<https://reviews.apache.org/r/40839/#comment168051>
Consider turning off perf tracing in the test runs (i.e. unit test runs). When perf-trace log is of interest, the logging can be enabled by simply uncommenting this block of perf-trace configs.
Move start of the comment at line #26 to line #11.
security-admin/src/test/resources/log4j.xml (line 26)
<https://reviews.apache.org/r/40839/#comment168053>
Earlier comment on another log4j.xml is applicable here as well. Please review.
- Madhan Neethiraj
On Dec. 1, 2015, 9:48 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40839/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2015, 9:48 p.m.)
>
>
> Review request for ranger and Madhan Neethiraj.
>
>
> Bugs: RANGER-753
> https://issues.apache.org/jira/browse/RANGER-753
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Instrumented policy download and policy evaluation for performance measurement.
> Optimized implementation of tag download.
> - Reduced number of database queries to fetch Tag-related objects. The number of database queries is now constant; it does not depend on the number of Tag-related objects in the ranger-admin.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerAbstractConditionEvaluator.java 0bcb744
> agents-common/src/main/java/org/apache/ranger/plugin/conditionevaluator/RangerConditionEvaluator.java 602b80e
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java b5662bf
> agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java ba2a5d7
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java 8adab7b
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 57b1b7d
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerAbstractPolicyItemEvaluator.java ffd1d79
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerCachedPolicyEvaluator.java d67777c
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyEvaluator.java da15c00
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerDefaultPolicyItemEvaluator.java 39a0a5e
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerOptimizedPolicyEvaluator.java e81280f
> agents-common/src/main/java/org/apache/ranger/plugin/policyevaluator/RangerPolicyItemEvaluator.java 95d2b4e
> agents-common/src/main/java/org/apache/ranger/plugin/resourcematcher/RangerAbstractResourceMatcher.java 56c4cfb
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractTagStore.java ed1b64d
> agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java ed20c51
> agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java 0729339
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java b59ae1f
> agents-common/src/test/resources/log4j.properties cb409e8
> agents-common/src/test/resources/log4j.xml PRE-CREATION
> security-admin/.gitignore bf7dc37
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 4c65178
> security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 300ba8d
> security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 8e2baab
> security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 04942a7
> security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 5ba3b74
> security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java c8cb91d
> security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java 28ddfde
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java d92fd41
> security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java b1986be
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4719fc9
> security-admin/src/test/resources/log4j.properties cb409e8
> security-admin/src/test/resources/log4j.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40839/diff/
>
>
> Testing
> -------
>
> Tested using python scripts to upload and download a large (of the order of tens of thousands) number of TagDefs, Tags, ServiceResources and TagResourceMap. Download of 10,000 tag-objects takes about 4 seconds in Ranger-Admin.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>