You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2021/11/19 02:25:18 UTC
Review Request 73715: RANGER-3519: Provide an option to optimize space needed by Trie objects
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73715/
-----------------------------------------------------------
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3519
https://issues.apache.org/jira/browse/RANGER-3519
Repository: ranger
Description
-------
When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
---
There are two new configuration variables that control the memory characteristics of Trie objects used by Ranger policy-engine.
<property-prefix>.policyengine.option.optimize.trie.for.space - if true, Trie objects for searching policies are optimized for space
<property-prefix>.policyengine.option.optimize.tag.trie.for.space - if true, Trie objects for searching tagged resources are optimized for space.
where <property-prefix> is "ranger.admin" - for policy engines used within Ranger Admin server, and it is "ranger.plugin.<service-type>" for policy engines used in the plug-in.
Diffs
-----
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java c8346d369
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java 07d0a3917
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 008ee7719
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerResourceTrie.java 7c37e05f9
Diff: https://reviews.apache.org/r/73715/diff/1/
Testing
-------
- Ran all unit tests successfully.
- Tested with a large (100K or more) number of Hive tagged resources, and verified that the heap size requirement is substantially reduced when "ranger.plugin.hive.policyengine.option.optimize.tag.trie.for.space" is set to true for HiveServer2 service.
Thanks,
Abhay Kulkarni
Re: Review Request 73715: RANGER-3519: Provide an option to optimize space needed by Trie objects
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73715/#review223753
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Nov. 19, 2021, 2:25 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73715/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2021, 2:25 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3519
> https://issues.apache.org/jira/browse/RANGER-3519
>
>
> Repository: ranger
>
>
> Description
> -------
>
> When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
>
> ---
>
> There are two new configuration variables that control the memory characteristics of Trie objects used by Ranger policy-engine.
>
> <property-prefix>.policyengine.option.optimize.trie.for.space - if true, Trie objects for searching policies are optimized for space
> <property-prefix>.policyengine.option.optimize.tag.trie.for.space - if true, Trie objects for searching tagged resources are optimized for space.
>
> where <property-prefix> is "ranger.admin" - for policy engines used within Ranger Admin server, and it is "ranger.plugin.<service-type>" for policy engines used in the plug-in.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java c8346d369
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java 07d0a3917
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 008ee7719
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerResourceTrie.java 7c37e05f9
>
>
> Diff: https://reviews.apache.org/r/73715/diff/1/
>
>
> Testing
> -------
>
> - Ran all unit tests successfully.
> - Tested with a large (100K or more) number of Hive tagged resources, and verified that the heap size requirement is substantially reduced when "ranger.plugin.hive.policyengine.option.optimize.tag.trie.for.space" is set to true for HiveServer2 service.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 73715: RANGER-3519: Provide an option to optimize space needed by Trie objects
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73715/#review223754
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Nov. 20, 2021, 3:55 a.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73715/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2021, 3:55 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3519
> https://issues.apache.org/jira/browse/RANGER-3519
>
>
> Repository: ranger
>
>
> Description
> -------
>
> When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
>
> ---
>
> There are two new configuration variables that control the memory characteristics of Trie objects used by Ranger policy-engine.
>
> <property-prefix>.policyengine.option.optimize.trie.for.space - if true, Trie objects for searching policies are optimized for space
> <property-prefix>.policyengine.option.optimize.tag.trie.for.space - if true, Trie objects for searching tagged resources are optimized for space.
>
> where <property-prefix> is "ranger.admin" - for policy engines used within Ranger Admin server, and it is "ranger.plugin.<service-type>" for policy engines used in the plug-in.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java c8346d369
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java 07d0a3917
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 008ee7719
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerResourceTrie.java 7c37e05f9
> distro/src/main/assembly/ranger-tools.xml 3ae890c1e
> ranger-tools/scripts/create_requests.py PRE-CREATION
> ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestConfiguration.java PRE-CREATION
> ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestEngine.java 37ac9ea74
> ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestOptions.java 0379fe6a6
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java 760ebb9ee
> ranger-tools/src/test/resources/testdata/ranger-config.xml 933d6705a
> ranger-tools/src/test/resources/testdata/test_requests_hive.json 0db7207ab
> ranger-tools/testdata/ranger-config.xml 933d6705a
>
>
> Diff: https://reviews.apache.org/r/73715/diff/3/
>
>
> Testing
> -------
>
> - Ran all unit tests successfully.
> - Tested with a large (100K or more) number of Hive tagged resources, and verified that the heap size requirement is substantially reduced when "ranger.plugin.hive.policyengine.option.optimize.tag.trie.for.space" is set to true for HiveServer2 service.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 73715: RANGER-3519: Provide an option to optimize space needed by Trie objects
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73715/
-----------------------------------------------------------
(Updated Nov. 20, 2021, 3:55 a.m.)
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
More updates to test tools
Bugs: RANGER-3519
https://issues.apache.org/jira/browse/RANGER-3519
Repository: ranger
Description
-------
When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
---
There are two new configuration variables that control the memory characteristics of Trie objects used by Ranger policy-engine.
<property-prefix>.policyengine.option.optimize.trie.for.space - if true, Trie objects for searching policies are optimized for space
<property-prefix>.policyengine.option.optimize.tag.trie.for.space - if true, Trie objects for searching tagged resources are optimized for space.
where <property-prefix> is "ranger.admin" - for policy engines used within Ranger Admin server, and it is "ranger.plugin.<service-type>" for policy engines used in the plug-in.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java c8346d369
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java 07d0a3917
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 008ee7719
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerResourceTrie.java 7c37e05f9
distro/src/main/assembly/ranger-tools.xml 3ae890c1e
ranger-tools/scripts/create_requests.py PRE-CREATION
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestConfiguration.java PRE-CREATION
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestEngine.java 37ac9ea74
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestOptions.java 0379fe6a6
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java 760ebb9ee
ranger-tools/src/test/resources/testdata/ranger-config.xml 933d6705a
ranger-tools/src/test/resources/testdata/test_requests_hive.json 0db7207ab
ranger-tools/testdata/ranger-config.xml 933d6705a
Diff: https://reviews.apache.org/r/73715/diff/3/
Changes: https://reviews.apache.org/r/73715/diff/2-3/
Testing
-------
- Ran all unit tests successfully.
- Tested with a large (100K or more) number of Hive tagged resources, and verified that the heap size requirement is substantially reduced when "ranger.plugin.hive.policyengine.option.optimize.tag.trie.for.space" is set to true for HiveServer2 service.
Thanks,
Abhay Kulkarni
Re: Review Request 73715: RANGER-3519: Provide an option to optimize space needed by Trie objects
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73715/
-----------------------------------------------------------
(Updated Nov. 20, 2021, 1:38 a.m.)
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Updated ranger-tools for performance testing
Bugs: RANGER-3519
https://issues.apache.org/jira/browse/RANGER-3519
Repository: ranger
Description
-------
When the number of policies (and/or tagged resources) is large, the data structures used by Ranger as indexes for policies (and/or tagged resources) may need a very large heap memory because they are optimized for fast lookup. It is desirable to be able to configure Ranger to have these structures optimized for space in order to keep the heap requirements within acceptable limit at the cost of somewhat slower lookup.
---
There are two new configuration variables that control the memory characteristics of Trie objects used by Ranger policy-engine.
<property-prefix>.policyengine.option.optimize.trie.for.space - if true, Trie objects for searching policies are optimized for space
<property-prefix>.policyengine.option.optimize.tag.trie.for.space - if true, Trie objects for searching tagged resources are optimized for space.
where <property-prefix> is "ranger.admin" - for policy engines used within Ranger Admin server, and it is "ranger.plugin.<service-type>" for policy engines used in the plug-in.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java c8346d369
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java 07d0a3917
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 008ee7719
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerResourceTrie.java 7c37e05f9
distro/src/main/assembly/ranger-tools.xml 3ae890c1e
ranger-tools/scripts/create_requests.py PRE-CREATION
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestConfiguration.java PRE-CREATION
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestEngine.java 37ac9ea74
ranger-tools/src/main/java/org/apache/ranger/policyengine/PerfTestOptions.java 0379fe6a6
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java 760ebb9ee
ranger-tools/testdata/ranger-config.xml 933d6705a
Diff: https://reviews.apache.org/r/73715/diff/2/
Changes: https://reviews.apache.org/r/73715/diff/1-2/
Testing
-------
- Ran all unit tests successfully.
- Tested with a large (100K or more) number of Hive tagged resources, and verified that the heap size requirement is substantially reduced when "ranger.plugin.hive.policyengine.option.optimize.tag.trie.for.space" is set to true for HiveServer2 service.
Thanks,
Abhay Kulkarni