You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2021/12/01 23:38:40 UTC
Re: Review Request 73728: RANGER-3519: Provide an option to optimize space needed by Trie objects - Part 2
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73728/
-----------------------------------------------------------
(Updated Dec. 1, 2021, 11:38 p.m.)
Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy.
Changes
-------
Updated with review comments
Bugs: RANGER-3519
https://issues.apache.org/jira/browse/RANGER-3519
Repository: ranger
Description
-------
When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
In addition to changed committed by (https://github.com/apache/ranger/commit/856571c4348e31725498c0922338339c76ebba02), following boolean valued configuration parameter is added for the Ranger admin server to optionally de-duplicate tags in ServiceTags object.
ranger.admin.supports.tags.dedup (default: false)
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerFileBasedTagRetriever.java b858879ba
agents-common/src/main/java/org/apache/ranger/plugin/service/RangerDefaultRequestProcessor.java facf05d43
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerCommonConstants.java 63bed50ce
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java 6b70b2259
agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java 9e8e0cf6c
distro/src/main/assembly/ranger-tools.xml 5fdf3a5ef
ranger-tools/scripts/create_requests.py 51efc0506
ranger-tools/scripts/gen_service_policies.sh PRE-CREATION
ranger-tools/scripts/gen_service_tags.sh dd3ee8813
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java f667628dd
ranger-tools/src/test/resources/testdata/ranger-config.xml 218db0759
ranger-tools/testdata/ranger-config.xml 08a83b9a5
security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 1bdfce057
Diff: https://reviews.apache.org/r/73728/diff/2/
Changes: https://reviews.apache.org/r/73728/diff/1-2/
Testing
-------
- Executed all unit tests successfully
- Performance tested ranger-tools with a million tags
Thanks,
Abhay Kulkarni
Re: Review Request 73728: RANGER-3519: Provide an option to optimize space needed by Trie objects - Part 2
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73728/#review223791
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Dec. 1, 2021, 11:38 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73728/
> -----------------------------------------------------------
>
> (Updated Dec. 1, 2021, 11:38 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3519
> https://issues.apache.org/jira/browse/RANGER-3519
>
>
> Repository: ranger
>
>
> Description
> -------
>
> When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
>
> In addition to changed committed by (https://github.com/apache/ranger/commit/856571c4348e31725498c0922338339c76ebba02), following boolean valued configuration parameter is added for the Ranger admin server to optionally de-duplicate tags in ServiceTags object.
>
> ranger.admin.supports.tags.dedup (default: false)
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerFileBasedTagRetriever.java b858879ba
> agents-common/src/main/java/org/apache/ranger/plugin/service/RangerDefaultRequestProcessor.java facf05d43
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerCommonConstants.java 63bed50ce
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java 6b70b2259
> agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java 9e8e0cf6c
> distro/src/main/assembly/ranger-tools.xml 5fdf3a5ef
> ranger-tools/scripts/create_requests.py 51efc0506
> ranger-tools/scripts/gen_service_policies.sh PRE-CREATION
> ranger-tools/scripts/gen_service_tags.sh dd3ee8813
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java f667628dd
> ranger-tools/src/test/resources/testdata/ranger-config.xml 218db0759
> ranger-tools/testdata/ranger-config.xml 08a83b9a5
> security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 1bdfce057
>
>
> Diff: https://reviews.apache.org/r/73728/diff/2/
>
>
> Testing
> -------
>
> - Executed all unit tests successfully
> - Performance tested ranger-tools with a million tags
>
>
> Thanks,
>
> Abhay Kulkarni
>
>