Re: Review Request 74090: RANGER-3852 performance and scalability analyzer tool for apache ranger

[Pradeep Agrawal <pr...@gmail.com>]

> On Aug. 19, 2022, 10:16 a.m., Pradeep Agrawal wrote:
> > Please fix the rat text/license issue. 
> > 
> > 4 Unknown Licenses
> > 
> > *******************************
> > 
> > Unapproved licenses:
> > 
> >   ranger-tools/src/main/python/requirements.txt
> >   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/__init__.py
> >   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_primary_config.json
> >   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_secondary_config.json

Attached the updated patch in the jira : https://issues.apache.org/jira/secure/attachment/13049578/0001-RANGER-3852-performance-and-scalability-analyzer-too.patch and will commit this patch. Please close this RR after that.


- Pradeep


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74090/#review224631
-----------------------------------------------------------


On Aug. 17, 2022, 4:54 a.m., Fateh Singh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74090/
> -----------------------------------------------------------
> 
> (Updated Aug. 17, 2022, 4:54 a.m.)
> 
> 
> Review request for ranger, bhavik patel, Dhaval Shah, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3852
>     https://issues.apache.org/jira/browse/RANGER-3852
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Simulation of user scenarios of high load or long duration of requests required to analyze performance of the system and detect abnormalities in case of high latency, memory, cpu usage etc
> 
> 
> Diffs
> -----
> 
>   ranger-tools/src/main/python/README.md PRE-CREATION 
>   ranger-tools/src/main/python/config/README.md PRE-CREATION 
>   ranger-tools/src/main/python/outputs/README.md PRE-CREATION 
>   ranger-tools/src/main/python/performance_analyzer.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/__init__.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/perf_globals.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/__init__.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_primary_config.json PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_secondary_config.json PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/path.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/__init__.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/base_object_stores.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/random_generators.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/service_object_stores.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/__init__.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/config_utils.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/dataframe_utils.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/extra_unused_utils.py PRE-CREATION 
>   ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/logging_utils.py PRE-CREATION 
>   ranger-tools/src/main/python/requirements.txt PRE-CREATION 
>   ranger-tools/src/main/python/setup_performance_analyzer.py PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/74090/diff/2/
> 
> 
> Testing
> -------
> 
> Aim:
> The tool has been developed for developers. 2 config files (primary and secondary) have been provided which the user can modify for their ranger environment including different api's and the number of calls for each api that need to be tested. Both API metrics (such as latency) as well as system metrics (such as CPU usage, memory usage etc) are obtained after the tool is run. The tool provides a framework with a lot of reusable components which can be used to extend the functionality to other API's which are not yet supported. 
> 
> Documentation and steps to run the tool:
> README.md file at the root level (ranger-tools/src/main/python directory) can be used as reference for the steps that need to be followed to run the tool. 
> The tool has been embedded with python docstrings which can be used to generate full documentation of the developed tool (details can be found in the README file). 
> 
> Currently supported API's for the tool: 
> ["create_policy", "update_policy_by_id", "get_policy_by_id", "delete_policy_by_id"]
> 
> Sanity checks and testing:
> The steps in the README.md file were followed in 2 separate instances independently to verify the correctness of the setup steps provided and sanity test of the tool developed.
> The tool has been tested with multiple number of calls and different api's ranging from 1-50 for different api's
> 
> 
> File Attachments
> ----------------
> 
> RANGER-3852-performance-and-scalability-analyzer-to.patch
>   https://reviews.apache.org/media/uploaded/files/2022/08/17/f750dde5-c39b-44b0-b293-ddf1e372c658__RANGER-3852-performance-and-scalability-analyzer-to.patch
> RANGER-3852-performance-and-scalability-analyzer-to.patch
>   https://reviews.apache.org/media/uploaded/files/2022/08/17/4a3e6441-1c60-4172-95a0-77d7b098deab__RANGER-3852-performance-and-scalability-analyzer-to.patch
> 
> 
> Thanks,
> 
> Fateh Singh
> 
>