You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ranger.apache.org by pr...@apache.org on 2022/09/21 14:40:28 UTC
[ranger] branch ranger-2.4 updated: RANGER-3852: performance and scalability analyzer tool for apache ranger
This is an automated email from the ASF dual-hosted git repository.
pradeep pushed a commit to branch ranger-2.4
in repository https://gitbox.apache.org/repos/asf/ranger.git
The following commit(s) were added to refs/heads/ranger-2.4 by this push:
new 97c508d99 RANGER-3852: performance and scalability analyzer tool for apache ranger
97c508d99 is described below
commit 97c508d99e9bb99282575d100ae4365d92dd8aa9
Author: Fateh Singh <fa...@gmail.com>
AuthorDate: Fri Aug 12 23:13:13 2022 -0700
RANGER-3852: performance and scalability analyzer tool for apache ranger
Signed-off-by: pradeep <pr...@apache.org>
---
pom.xml | 4 +
ranger-tools/src/main/python/README.md | 95 +++++++
ranger-tools/src/main/python/config/README.md | 94 +++++++
ranger-tools/src/main/python/outputs/README.md | 27 ++
.../src/main/python/performance_analyzer.py | 237 +++++++++++++++++
.../python/ranger_performance_tool/__init__.py | 17 ++
.../python/ranger_performance_tool/perf_globals.py | 48 ++++
.../ranger_perf_assets/__init__.py | 17 ++
.../ranger_perf_assets/default_primary_config.json | 38 +++
.../default_secondary_config.json | 58 ++++
.../ranger_perf_assets/path.py | 21 ++
.../ranger_perf_object_stores/__init__.py | 17 ++
.../base_object_stores.py | 237 +++++++++++++++++
.../ranger_perf_object_stores/random_generators.py | 188 +++++++++++++
.../service_object_stores.py | 189 +++++++++++++
.../ranger_perf_utils/__init__.py | 0
.../ranger_perf_utils/config_utils.py | 85 ++++++
.../ranger_perf_utils/dataframe_utils.py | 129 +++++++++
.../ranger_perf_utils/extra_unused_utils.py | 99 +++++++
.../ranger_perf_utils/logging_utils.py | 293 +++++++++++++++++++++
ranger-tools/src/main/python/requirements.txt | 22 ++
.../src/main/python/setup_performance_analyzer.py | 55 ++++
22 files changed, 1970 insertions(+)
diff --git a/pom.xml b/pom.xml
index 570227ce2..eaa4f6d65 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1161,6 +1161,10 @@
<exclude>**/package-lock.json</exclude>
<exclude>**/ranger_es_schema.json</exclude>
<exclude>**/venv/**</exclude>
+ <exclude>**/requirements.txt</exclude>
+ <exclude>**/__init__.py</exclude>
+ <exclude>**/default_primary_config.json</exclude>
+ <exclude>**/default_secondary_config.json</exclude>
</excludes>
</configuration>
</plugin>
diff --git a/ranger-tools/src/main/python/README.md b/ranger-tools/src/main/python/README.md
new file mode 100644
index 000000000..4763a1d51
--- /dev/null
+++ b/ranger-tools/src/main/python/README.md
@@ -0,0 +1,95 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Performance and Scalability Analyzer for Apache Ranger
+## Documentation
+Run the below command to generate pydocs for the package. Code base has doc strings describing the methods and classes from which the document is generated.
+
+```bash
+> python -m pydoc -b
+```
+
+Other README files can be found in the following directory:
+
+```/config/README.md```: Describes the primary and secondary config files which the user has to modify
+
+```/outputs/README.md```: Describes the output files generated by the analyzer
+
+## Server side installation
+```bash
+> sudo apt-get install sysstat
+or
+> sudo yum install sysstat
+```
+```ranger.accesslog.pattern``` to include the `%D` in the access pattern so that the tomcat server also logs the api execution time
+
+## Client side Installation
+
+Use the package manager [pip](https://pip.pypa.io/en/stable/) to install requirements for running the performance tests.
+Ensure right path to requirements.txt is given.
+
+```bash
+> pip install -r requirements.txt
+
+> apt-get install sshpass
+or
+> brew install hudochenkov/sshpass/sshpass
+```
+
+
+## Usage
+```cd``` into ```python``` directory before executing below commands
+
+First time usage or to reset the config files:
+```bash
+> python setup_performance_analyzer.py
+```
+
+Subsequent usage:
+Fill out the config details (```primary_config.json``` and ```secondary_config.json```) in ```/config``` folder or keep default
+
+For multiple api testing (Uses values from config file).
+
+usage:
+
+```bash
+> python3 performance_analyzer.py
+```
+
+For single api testing (Command line arguments override config file values)
+
+usage:
+
+```bash
+> python performance_analyzer.py --ranger_url <ranger_url> --calls <number of times to call api> --api <name of function of python client corresponding to api> --username <Auth username> --password <Auth password> --client_ip <client ip address> --ssh_host <ranger host to connect for ssh> --ssh_user <Server user e.g. root> --ssh_password <Server password>
+```
+
+Example command:
+
+```bash
+> python3 performance_analyzer.py --ranger_url http://ranger_host:ranger_admin_port --calls 10 --api create_policy --username admin --password Admin123 --client_ip vpn_ip_client_in_logs --ssh_host ranger_host --ssh_user ssh_user --ssh_password ssh_password
+```
+
+## Description of tools used
+System metrics on server side are collected using [vmstat](https://phoenixnap.com/kb/vmstat-command)
+
+## Warnings
+Ensure sudo/root privileges for the user on the server side for vmstat command.
+
+Ensure VPN is enables and client can communicate with the server. Else, in some cases stale values from previous successful run of the tool may be presented
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/config/README.md b/ranger-tools/src/main/python/config/README.md
new file mode 100644
index 000000000..d8f09c27c
--- /dev/null
+++ b/ranger-tools/src/main/python/config/README.md
@@ -0,0 +1,94 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Performance and Scalability Analyzer for Apache Ranger
+
+## Primary Config Description
+1) List of apis to be tested for performance and scalability. Currently only below API's supported.
+
+`"api_list": [
+ "create_policy",
+ "update_policy_by_id",
+ "get_policy_by_id",
+ "delete_policy_by_id"
+ ]`
+
+2) SSH username and password for the machine where Ranger is running:
+`user` and `password`
+
+3) `client_ip`: for access log parsing
+4) Config for system metrics collection. Runs `vmstat` command in the backend machine and parses the output.
+`"system_logger": {
+ "enabled": true,
+ "align_with_access_logs": false,
+ "metrics": [
+ "r",
+ "free",
+ "id",
+ "UTC"
+ ],
+ "sleep_seconds": 5,
+ "num_calls": "inf",
+ "remote_log_file_location": "xxxxxxxx/system_logs.txt",
+ "secondary_log_file_location": "xxxxxxxxx/secondary_system_logs.txt"
+ }`
+
+```enabled:false``` may break the currently implemented system
+
+```align_with_access_logs:true``` aligns the access logs and system logs by timestamp
+
+```sleep_seconds:5``` sleep time between system metrics collection
+
+```num_calls:inf``` number of system metrics collection. inf means infinite collection till the end of the performance_analyzer.py script execution (if the script crashes, then manual killing of vmstat may be required).
+
+```remote_log_file_location: xxxxxxx/system_logs.txt``` location of system logs. default ok for most cases.
+
+```secondary_log_file_location: xxxxxxxxxx/secondary_system_logs.txt``` location of secondary system logs. default ok for most cases.
+
+```metrics: "r","free","id","UTC"]```metrics to be collected. default ok for most cases. Full list of supported metrics: keys of dict header_mapping_system_logs in ```ranger_performance_tool/ranger_perf_utils/logging_utils.py```
+
+5) API's currently supported. Change num_calls and sleep_seconds as per the load and benchmarking needs.
+```
+"api": {
+ "create_policy": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "delete_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "get_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "update_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ }
+ }
+```
+
+## Secondary Config Description
+1) "random_type" is a random number generator type for differeny generators implemented in ```ranger_performance_tool/ranger_perf_object_stores/random_generators.py```.
+Currently implemented and supported "random_type" is one of `["random", "increasing", "incremental"]`.
+Check ```ranger_performance_tool/ranger_perf_object_stores/random_generators.py``` for more details or add new custom generators.
+
+2) "enabled_services" is a list of services to be enabled for the tests.
+3) "service_type_mapping" is a mapping of service name to service type.
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/outputs/README.md b/ranger-tools/src/main/python/outputs/README.md
new file mode 100644
index 000000000..38ad164f4
--- /dev/null
+++ b/ranger-tools/src/main/python/outputs/README.md
@@ -0,0 +1,27 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Performance and Scalability Analyzer for Apache Ranger
+
+## Output files description
+1) performance_report.html: Open in browser to view the performance report
+2) performance_report.csv: CSV file with performance report
+3) performance_report.json: JSON file with performance report
+4) statistics_report.json: JSON file with statistics report - contains crunched numbers for different metrics
+5) statistics_report.csv: CSV file with statistics report - contains crunched numbers for different metrics
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/performance_analyzer.py b/ranger-tools/src/main/python/performance_analyzer.py
new file mode 100644
index 000000000..e437d77d8
--- /dev/null
+++ b/ranger-tools/src/main/python/performance_analyzer.py
@@ -0,0 +1,237 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+For single api testing. Command line arguments override config file values.
+usage:
+ python performance_analyzer.py --ranger_url <ranger_url> --calls <number of times to call api> --api
+ <name of function of apache_ranger python client corresponding to api> --username <Auth username>
+ --password <Auth password> --client_ip <client ip address> --ssh_host <ranger host to connect for ssh>
+ --ssh_user <Server user e.g. root> --ssh_password <Server password>
+Example command:
+ python3 performance_analyzer.py --ranger_url http://ranger_host:6080
+ --calls 10 --api create_policy --username ranger_admin --password ranger_password
+ --client_ip vpn_ip --ssh_host ranger_host
+ --ssh_user ssh_username --ssh_password ssh_password
+
+For multiple api testing. Uses values from config file.
+usage:
+ python3 performance_analyzer.py
+
+
+"""
+import time
+import logging
+import sys
+import argparse
+
+import pandas as pd
+import seaborn as sns
+
+import ranger_performance_tool.perf_globals as perf_globals
+from ranger_performance_tool.ranger_perf_utils.logging_utils import SystemLogger, LogFetcher, LogParser
+from ranger_performance_tool.ranger_perf_utils.dataframe_utils import DataframeUtils
+
+
+def performance_analyzer_main(argv_dict):
+
+ configparser = perf_globals.CONFIG_READER
+
+ object_store = perf_globals.OBJECT_STORE
+
+ if len(argv_dict.keys()) != 0:
+ configparser.override_with_command_line_args(argv_dict)
+
+ clear = configparser.get_config_value("primary", "clear")
+ host = configparser.get_config_value("primary", "host_name")
+ user = configparser.get_config_value("primary", "user")
+ password = configparser.get_config_value("primary", "password")
+
+ sys_logger = None
+ if configparser.get_config_value("primary", "system_logger", "enabled"):
+ log_file = configparser.get_config_value("primary", "system_logger", "remote_log_file_location")
+ secondary_log_file = configparser.get_config_value("primary", "system_logger", "secondary_log_file_location")
+ sys_logger = SystemLogger(host, user, password, log_file, secondary_log_file)
+
+ if clear:
+ if sys_logger:
+ sys_logger.delete_old_logs()
+
+ if sys_logger:
+ sys_logger.start_system_log_service(configparser.get_config_value("primary", "system_logger", "sleep_seconds"),
+ configparser.get_config_value("primary", "system_logger", "num_calls"))
+ sys_logger.execute_secondary_system_log_command()
+
+ ranger = perf_globals.RANGER_CLIENT
+
+ api_list = configparser.get_config_value("primary", "api_list")
+ for api in api_list:
+ num_calls = configparser.get_config_value("primary", "api", api, "num_calls")
+ sleep_seconds = configparser.get_config_value("primary", "api", api, "sleep_seconds")
+ for i in range(num_calls):
+ try:
+ params = object_store.get_api_param_dict(api)
+ print(api, i, params)
+ resp = object_store.get_api(ranger, api)(**params)
+ time.sleep(sleep_seconds)
+ print(resp)
+ except Exception as e:
+ print(e)
+
+ df_utils = DataframeUtils()
+
+ log_fetcher = LogFetcher()
+
+ log_parser = LogParser()
+
+ access_log_file = log_fetcher.fetch_access_logs_from_server(host, user, password,
+ remote_path=configparser.get_config_value(
+ "primary",
+ "remote_access_log_location"
+ ))
+ access_df = log_parser.parse_access_logs(access_log_file, configparser.get_config_value("primary", "client_ip"))
+ print(access_df.to_string())
+
+ if sys_logger:
+ sys_logger.stop_system_log_service()
+ sys_log_file = log_fetcher.fetch_system_logs_from_server(
+ host, user, password, configparser.get_config_value("primary", "system_logger", "remote_log_file_location"))
+ secondary_sys_log_file = log_fetcher.fetch_secondary_system_logs_from_server(
+ host, user, password, configparser.get_config_value("primary", "system_logger",
+ "secondary_log_file_location"))
+ metrics = configparser.get_config_value("primary", "system_logger", "metrics")
+ if len(metrics) > 0:
+ main_system_df = log_parser.parse_system_logs(sys_log_file, metrics)
+ else:
+ main_system_df = log_parser.parse_system_logs(sys_log_file)
+
+ print("system stats = \n",main_system_df.to_string())
+ secondary_sys_log_df = log_parser.parse_secondary_system_logs(secondary_sys_log_file)
+ system_df = df_utils.combine_system_logs_dataframe(main_system_df, secondary_sys_log_df)
+
+ align_with_access_logs = configparser.get_config_value("primary", "system_logger", "align_with_access_logs")
+
+ if align_with_access_logs:
+ # join and align based on time
+ aligned_df = df_utils.align_dataframes(system_df, access_df, system_logs_timestamp_col_name='UTC',
+ access_logs_timestamp_col_name='time', merge=True)
+ df_utils.rename_columns(aligned_df, LogParser.header_mapping_system_logs)
+ print(aligned_df.to_string())
+
+ statistics_df = aligned_df.describe()
+ df_utils.rename_rows(statistics_df, {"25%": "25th_percentile", "50%": "median", "75%": "75th_percentile"})
+
+ num_api_calls_received_at_server = aligned_df[~aligned_df['ip'].isnull()].shape[0]
+ num_api_calls_sent_to_server = sum([configparser.get_config_value("primary", "api", api, "num_calls") for api in api_list])
+ df_utils.insert_column(statistics_df, "num_api_calls_sent_to_server", num_api_calls_sent_to_server)
+ df_utils.insert_column(statistics_df, "num_api_calls_received_at_server", num_api_calls_received_at_server)
+ with open("statistics_report.csv", "w") as f:
+ statistics_df.to_csv(f)
+
+ with open("performance_report.csv", "w") as f:
+ aligned_df.to_csv(f, index=False)
+
+ with open("performance_report.html", "w") as f:
+ cm = sns.light_palette("red", as_cmap=True)
+ html = aligned_df.style.background_gradient(cmap=cm).to_html()
+ f.write(html)
+ else:
+ aligned_df = df_utils.align_dataframes(system_df, access_df, system_logs_timestamp_col_name='UTC',
+ access_logs_timestamp_col_name='time', merge=False)
+ print(aligned_df.to_string())
+
+ statistics_df_access = aligned_df.describe()
+ statistics_df_system = system_df.describe()
+
+ statistics_df = pd.concat([statistics_df_access, statistics_df_system], axis=1)
+ df_utils.rename_rows(statistics_df, {"25%": "25th_percentile", "50%": "median", "75%": "75th_percentile"})
+
+ num_api_calls_received_at_server = aligned_df[~aligned_df['ip'].isnull()].shape[0]
+ num_api_calls_sent_to_server = sum([configparser.get_config_value("primary", "api", api, "num_calls") for api in api_list])
+
+ df_utils.insert_column(statistics_df, "num_api_calls_sent_to_server", num_api_calls_sent_to_server)
+ df_utils.insert_column(statistics_df, "num_api_calls_received_at_server", num_api_calls_received_at_server)
+
+ df_utils.rename_columns(statistics_df, log_parser.header_mapping_system_logs)
+
+ with open(perf_globals.OUTPUT_DIR+"statistics_report.json", "w") as f:
+ statistics_df.to_json(f)
+
+ with open(perf_globals.OUTPUT_DIR+"statistics_report.csv", "w") as f:
+ statistics_df.to_csv(f)
+
+ with open(perf_globals.OUTPUT_DIR+"performance_report.json", "w") as f:
+ aligned_df.to_json(f)
+
+ with open(perf_globals.OUTPUT_DIR+"performance_report.csv", "w") as f:
+ aligned_df.to_csv(f, index=False)
+
+ with open(perf_globals.OUTPUT_DIR+"performance_report.html", "w") as f:
+ cm = sns.light_palette("red", as_cmap=True)
+ html = aligned_df.style.background_gradient(cmap=cm).to_html()
+ f.write(html)
+
+
+def log(msg, type):
+ if type == 'info':
+ logging.info(" %s", msg)
+ if type == 'debug':
+ logging.debug(" %s", msg)
+ if type == 'warning':
+ logging.warning(" %s", msg)
+ if type == 'exception':
+ logging.exception(" %s", msg)
+ if type == 'error':
+ logging.error(" %s", msg)
+
+
+def print_usage():
+ print("usage:: python performance_analyzer.py --ranger_url <ranger_url> --calls <number of times to call api> --api"
+ " <name of function of python client corresponding to api> --username <Auth username> "
+ "--password <Auth password> --client_ip <client ip address> --ssh_host <ranger host to connect for ssh> "
+ "--ssh_user <Server user e.g. root> --ssh_password <Server password>")
+
+
+def main(argv):
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--ranger_url", help="ranger url")
+ parser.add_argument("--calls", help="number of times to call api")
+ parser.add_argument("--api", help="name of function of python client corresponding to api")
+ parser.add_argument("--username", help="Auth username")
+ parser.add_argument("--password", help="Auth password")
+ parser.add_argument("--client_ip", help="client ip address")
+ parser.add_argument("--ssh_host", help="ranger host to connect for ssh")
+ parser.add_argument("--ssh_user", help="Server user e.g. root")
+ parser.add_argument("--ssh_password", help="Server password")
+ ns = parser.parse_args(argv)
+ commandline_argument_dict = vars(ns)
+ try:
+ if None in commandline_argument_dict.values() and all(i is None for i in commandline_argument_dict.values()):
+ commandline_argument_dict = {}
+ if None in commandline_argument_dict.values() and not all(i is None for i in commandline_argument_dict.values()):
+ commandline_argument_dict = {}
+ print_usage()
+ raise ValueError("Either all the commandline arguments are provided or none are provided to run the script."
+ "Ignoring the provided arguments and reading from the config files.Continuing with execution.\n")
+ except ValueError as e:
+ print(e)
+
+ performance_analyzer_main(commandline_argument_dict)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/__init__.py b/ranger-tools/src/main/python/ranger_performance_tool/__init__.py
new file mode 100644
index 000000000..ed9d0b3c4
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/__init__.py
@@ -0,0 +1,17 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/perf_globals.py b/ranger-tools/src/main/python/ranger_performance_tool/perf_globals.py
new file mode 100644
index 000000000..7708c6e70
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/perf_globals.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+global variables for performance tool.
+creates repository to reusable objects.
+Use these variables to access configs, object stores, ranger client etc.
+"""
+
+import os
+
+from apache_ranger.client.ranger_client import RangerClient
+
+from ranger_performance_tool.ranger_perf_utils.config_utils import ConfigReader
+from ranger_performance_tool.ranger_perf_object_stores.base_object_stores import RangerAPIObjectStore
+
+BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), '../'))
+
+CONFIG_DIR = os.path.join(BASE_DIR, 'config/')
+
+PRIMARY_CONFIG_PATH = os.path.join(CONFIG_DIR, 'primary_config.json')
+
+SECONDARY_CONFIG_PATH = os.path.join(CONFIG_DIR, 'secondary_config.json')
+
+CONFIG_READER = ConfigReader(PRIMARY_CONFIG_PATH, SECONDARY_CONFIG_PATH)
+
+RANGER_CLIENT = RangerClient(CONFIG_READER.get_config_value("primary", "ranger_url"),
+ tuple(CONFIG_READER.get_config_value("primary", "ranger_auth")))
+
+OBJECT_STORE = RangerAPIObjectStore()
+
+OUTPUT_DIR = os.path.join(BASE_DIR, 'outputs/')
+
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/__init__.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/__init__.py
new file mode 100644
index 000000000..f7298b5f1
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/__init__.py
@@ -0,0 +1,17 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_primary_config.json b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_primary_config.json
new file mode 100644
index 000000000..322304ace
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_primary_config.json
@@ -0,0 +1,38 @@
+{
+ "clear": true,
+ "ranger_url": "http://ranger_host:XX_port_XX/",
+ "ranger_auth": ["X_user_name_X", "XX_password_XXX"],
+ "host_name": "ranger_host",
+ "user": "xxxx_ssh_username_xxxx",
+ "password": "xxxxx_ssh_password_xxxxx",
+ "client_ip": "xx.xx.xx.xx (IP address of the client used for logging (e.g. Client IP for VPN) )",
+ "api_list": ["create_policy", "update_policy_by_id", "get_policy_by_id", "delete_policy_by_id"],
+ "remote_access_log_location": "/xxxxxx/logs/",
+ "system_logger": {
+ "enabled": true,
+ "align_with_access_logs": false,
+ "metrics": ["r", "free", "id", "UTC"],
+ "sleep_seconds": 5,
+ "num_calls": "inf",
+ "remote_log_file_location": "xxxxxxxx/system_logs.txt",
+ "secondary_log_file_location": "xxxxxxxxx/secondary_system_logs.txt"
+ },
+ "api": {
+ "create_policy": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "delete_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "get_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ },
+ "update_policy_by_id": {
+ "num_calls": 3,
+ "sleep_seconds": 1
+ }
+ }
+}
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_secondary_config.json b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_secondary_config.json
new file mode 100644
index 000000000..6c91ffb9c
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/default_secondary_config.json
@@ -0,0 +1,58 @@
+{
+ "enabled_services": ["hdfs_service", "hive_service", "kafka_service"],
+ "service_type_mapping": {
+ "hdfs_service": "hdfs",
+ "hive_service": "hive",
+ "hbase_service": "hbase",
+ "yarn_service": "yarn",
+ "kafka_service": "kafka",
+ "knox_service": "knox",
+ "solr_service": "solr",
+ "atlas_service": "atlas",
+ "ozone_service": "ozone",
+ "adls_service": "adls",
+ "s3_service": "s3",
+ "kudu_service": "kudu"
+ },
+ "policy_store": {
+ "random_type": "random"
+ },
+ "services": {
+ "hdfs": {
+ "random_type": "random"
+ },
+ "hbase": {
+ "random_type": "random"
+ },
+ "hive": {
+ "random_type": "random"
+ },
+ "yarn": {
+ "random_type": "random"
+ },
+ "knox": {
+ "random_type": "random"
+ },
+ "solr": {
+ "random_type": "random"
+ },
+ "kafka": {
+ "random_type": "random"
+ },
+ "atlas": {
+ "random_type": "random"
+ },
+ "ozone": {
+ "random_type": "random"
+ },
+ "adls": {
+ "random_type": "random"
+ },
+ "s3": {
+ "random_type": "random"
+ },
+ "kudu": {
+ "random_type": "random"
+ }
+ }
+}
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/path.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/path.py
new file mode 100644
index 000000000..cfa856b14
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_assets/path.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), '../'))
+DEFAULT_PRIMARY_CONFIG_PATH = os.path.join(BASE_DIR, 'ranger_perf_assets/default_primary_config.json')
+DEFAULT_SECONDARY_CONFIG_PATH = os.path.join(BASE_DIR, 'ranger_perf_assets/default_secondary_config.json')
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/__init__.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/__init__.py
new file mode 100644
index 000000000..f7298b5f1
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/__init__.py
@@ -0,0 +1,17 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/base_object_stores.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/base_object_stores.py
new file mode 100644
index 000000000..efc161662
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/base_object_stores.py
@@ -0,0 +1,237 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from apache_ranger.model.ranger_policy import *
+from ranger_performance_tool.ranger_perf_object_stores.service_object_stores import *
+from ranger_performance_tool.ranger_perf_object_stores.random_generators import *
+
+
+class RangerAPIObjectStore:
+ """
+ Primary class to connect object stores to the Ranger API
+
+ Attributes
+ ----------
+ service_store : dict
+ Mapping of service type to service store objects. Add new service names here if required or unsupported.
+ policy_store: PolicyStore object to create objects associated with policy api's.
+
+ Methods
+ -------
+ get_service_type_for_service_name(service_name)
+ Returns the service type for the given service name
+ get_service_store(service_type)
+ Returns the service store for the given service type
+ get_api(ranger_client, api_name)
+ Returns the API object for the given API name
+ get_api_param_dict(api_name)
+ Returns the parameter dictionary for the given API name
+ """
+ service_store_def = {
+ "hdfs": HDFSServiceStore,
+ "hbase": HBaseServiceStore,
+ "hive": HiveServiceStore,
+ "yarn": YarnServiceStore,
+ "knox": KnoxServiceStore,
+ "solr": SolrServiceStore,
+ "kafka": KafkaServiceStore,
+ "atlas": AtlasServiceStore,
+ "ozone": OzoneServiceStore,
+ "adls": AdlsServiceStore,
+ "s3": S3ServiceStore,
+ "kudu": KuduServiceStore,
+ }
+
+ def __init__(self):
+ """
+ Constructor
+ Modify if new object stores are added
+ """
+ self.policy_store = PolicyStore()
+ self.service_store = {}
+
+ from ranger_performance_tool import perf_globals
+ enabled_services = perf_globals.CONFIG_READER.get_config_value("secondary", "enabled_services")
+ service_type_mapping = perf_globals.CONFIG_READER.get_config_value("secondary", "service_type_mapping")
+ for service_name in enabled_services:
+ if service_name not in service_type_mapping:
+ raise Exception(f"Unknown service name:{service_name}. "
+ f"Add it to service_type_mapping in secondary config file")
+ service_type = service_type_mapping[service_name]
+ random_type = perf_globals.CONFIG_READER.get_config_value("secondary", "services",
+ service_type, "random_type")
+ self.service_store[service_type] = RangerAPIObjectStore.service_store_def[service_type](random_type=random_type)
+
+ def get_service_type_for_service_name(self, service_name):
+ """
+ Returns the service type for the given service name
+ :parameter service_name: Service name for the service type to be returned
+ :return: Service type for the given service name. See service_type_mapping dictionary for supported service names.
+ """
+ from ranger_performance_tool import perf_globals
+ service_type_mapping = perf_globals.CONFIG_READER.get_config_value("secondary", "service_type_mapping")
+ if service_name not in service_type_mapping.keys():
+ raise Exception(f"Unknown service name:{service_name}."
+ f"Add it to service_type_mapping in secondary config file")
+ return service_type_mapping[service_name]
+
+ def get_service_store(self, service_type):
+ """
+ Returns the service store for the given service type
+ :parameter service_type: Service type for the service store to be returned
+ :return: Service store object for the given service type. See service_store dictionary for supported service types.
+ """
+ if service_type not in self.service_store:
+ raise Exception("Service name not found in service store")
+ return self.service_store[service_type]
+
+ def get_api(self, ranger_client, api_name):
+ """
+ Returns the API object for the given API name
+ :parameter ranger_client: Ranger client object
+ :parameter api_name: Name of the method corresponding to intg python ranger client
+ :return: API callable object
+ """
+ if api_name == "create_policy":
+ return ranger_client.create_policy
+ elif api_name == "delete_policy_by_id":
+ return ranger_client.delete_policy_by_id
+ elif api_name == "get_policy_by_id":
+ return ranger_client.get_policy_by_id
+ elif api_name == "update_policy_by_id":
+ return ranger_client.update_policy_by_id
+ else:
+ raise Exception(f"Unknown API name: {api_name}")
+
+ def get_api_param_dict(self, api_name):
+ """
+ Returns the parameter dictionary for the given API name
+ :parameter api_name: Name of the method corresponding to intg python ranger client
+ :return: Parameter dictionary to be used in conjunction with the API callable object from get_api()
+ """
+ if api_name == "create_policy":
+ param_dict = {
+ "policy": self.policy_store.generate_object()
+ }
+ elif api_name == "delete_policy_by_id":
+ param_dict = {
+ "policyId": self.policy_store.get_id()
+ }
+ elif api_name == "get_policy_by_id":
+ param_dict = {
+ "policyId": self.policy_store.get_id()
+ }
+ elif api_name == "update_policy_by_id":
+ policyId = self.policy_store.get_id()
+ policy = self.policy_store.get_policy_by_id(policyId)
+ param_dict = {
+ "policyId": policyId,
+ "policy": self.policy_store.generate_object(id=policyId, service=policy.service,
+ serviceType=policy.serviceType)
+ }
+ else:
+ raise Exception(f"Unknown API name: {api_name}")
+
+ return param_dict
+
+
+class RemoteStore:
+ """
+ Calls RangerClient api's to support other object stores. For example see usage in PolicyStore.
+ """
+ def __init__(self):
+ from ranger_performance_tool import perf_globals
+ self.ranger_client = perf_globals.RANGER_CLIENT
+ self.policy_list = None
+
+ def get_policy_list(self):
+ if self.policy_list is None:
+ from ranger_performance_tool import perf_globals
+ self.policy_list = {service: self.ranger_client.get_policies_in_service(service)
+ for service in perf_globals.CONFIG_READER.get_config_value("secondary",
+ "enabled_services")}
+ return self.policy_list
+
+
+class PolicyStore:
+ """
+ Support object creation for API's related to policies
+
+ Attributes:
+ random_generator: Type of random generator based on random_type defines in secondary config file
+ remote_store: Remote store object that fetches required policy objects from RangerClient
+
+ Methods:
+ generate_object: Generates a policy object with random values base on random_generator type
+ get_id: Returns the id of the policy object by looking up the policy object in the remote store
+ get_policy_by_id: Returns the policy object by looking up the policy object in the remote store
+ """
+ def __init__(self):
+ from ranger_performance_tool import perf_globals
+ self.random_generator = random_generators.get_random_generator(
+ perf_globals.CONFIG_READER.get_config_value("secondary", "policy_store", "random_type"))
+ self.remote_store = RemoteStore()
+
+ def get_policy_by_id(self, id):
+ """
+ Returns the policy object by looking up the policy object in the remote store
+ :param id: int , id of the policy object
+ :return: policy: dict , policy object
+ """
+ for service, policy_list in self.remote_store.get_policy_list().items():
+ for policy in policy_list:
+ if policy.id == id:
+ return policy
+
+ def get_id(self):
+ """
+ Returns the id of the policy object by looking up the policy object in the remote store
+ :return: id: int , id of the policy object
+ """
+ from ranger_performance_tool import perf_globals
+ enabled_services = perf_globals.CONFIG_READER.get_config_value("secondary", "enabled_services")
+ service = random.choice(enabled_services)
+ policy_list = self.remote_store.get_policy_list()[service]
+ return random.choice(policy_list).id
+
+ def generate_object(self, **kwargs):
+ """
+ Generates a policy object with random values base on random_generator type
+ :param kwargs: dict , optional parameters to be used in generating the policy object. Overrides the default generated values.
+ :return: policy_object: dict , policy object
+ """
+ from ranger_performance_tool import perf_globals
+ service_type_mapping = perf_globals.CONFIG_READER.get_config_value("secondary", "service_type_mapping")
+ if "id" not in kwargs:
+ kwargs["id"] = self.random_generator.generate_int()
+ if "name" not in kwargs:
+ kwargs["name"] = self.random_generator.generate_string()
+ if "service" not in kwargs:
+ enabled_service = random.choice(perf_globals.CONFIG_READER.get_config_value("secondary", "enabled_services"))
+ service_type = service_type_mapping[enabled_service]
+ kwargs["service"] = enabled_service
+ kwargs["serviceType"] = service_type
+ policy_object = RangerPolicy()
+ for key, value in kwargs.items():
+ if key not in dir(policy_object):
+ raise Exception("Invalid key: " + key)
+ policy_object[key] = value
+ service_type = policy_object.serviceType
+ service_store = perf_globals.OBJECT_STORE.service_store[service_type]
+ policy_object.resources = service_store.generate_resources()
+ return policy_object
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/random_generators.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/random_generators.py
new file mode 100644
index 000000000..047563226
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/random_generators.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import random
+import string
+
+
+def get_random_generator(random_type):
+ """
+ Returns the random generator based on random_type defines in secondary config file
+ :param random_type: str , type of random generator, supported values are "random", "random_int", "random_string"
+ :return: generator: object , random generator object corresponding to random_type
+ """
+ if random_type == "increasing":
+ generator = RandomGeneratorIncreasingSize()
+ elif random_type == "random":
+ generator = RandomGeneratorRandom()
+ elif random_type == "incremental":
+ generator = RandomGeneratorIncremental()
+ else:
+ raise Exception(f"Invalid type of Random Generator: {type}")
+ return generator
+
+
+class RandomGeneratorIncreasingSize:
+ """
+ One of the supported generators with size of object increasing with each call to generator functions
+
+ Attributes
+ ----------
+ int_counter : int
+ counter for generating int values
+ string_length : int
+ length of the string to be generated
+ array_length : int
+ length of the array to be generated
+
+ Methods
+ -------
+ generate_int()
+ generates int value
+ generate_string()
+ generates string value of length string_length
+ generate_array()
+ generates array of length array_length
+ generate_string_list_2d(num_rows)
+ generates 2d array of strings of size num_rows x array_length
+ generate_string_list_2d(num_rows, num_cols)
+ generates 2d array of strings of size num_rows x num_cols
+ """
+
+ LARGE_CONST = 100
+
+ def __init__(self):
+ self.int_counter = 0
+ self.string_length = 1
+ self.array_length = 1
+
+ def generate_int(self):
+ self.int_counter += 1
+ return self.int_counter
+
+ def generate_string(self):
+ self.string_length += RandomGeneratorIncreasingSize.LARGE_CONST
+ return ''.join(random.choices(string.ascii_letters, k=self.string_length))
+
+ def generate_array(self):
+ self.array_length += RandomGeneratorIncreasingSize.LARGE_CONST
+ return [self.generate_string() for _ in range(self.array_length)]
+
+ def generate_string_list_2d(self, num_rows):
+ self.array_length += RandomGeneratorIncreasingSize.LARGE_CONST
+ return [[self.generate_string() for _ in range(self.array_length)] for _ in range(num_rows)]
+
+ def generate_string_list_2d(self, num_rows, num_cols):
+ self.array_length += RandomGeneratorIncreasingSize.LARGE_CONST
+ return [[self.generate_string() for _ in range(num_cols)] for _ in range(num_rows)]
+
+
+class RandomGeneratorIncremental:
+ """
+ One of the supported generators with size of object increasing by 1 with each call to generator functions
+
+ Attributes
+ ----------
+ int_counter : int
+ counter for generating int values
+ string_length : int
+ length of the string to be generated
+ array_length : int
+ length of the array to be generated
+
+ Methods
+ -------
+ generate_int()
+ generates int value
+ generate_string()
+ generates string value of length string_length
+ generate_array()
+ generates array of length array_length
+ generate_string_list_2d(num_rows)
+ generates 2d array of strings of size num_rows x array_length
+ generate_string_list_2d(num_rows, num_cols)
+ generates 2d array of strings of size num_rows x num_cols
+ """
+ def __init__(self):
+ self.int_counter = 0
+ self.string_length = 1
+ self.array_length = 1
+
+ def generate_int(self):
+ self.int_counter += 1
+ return self.int_counter
+
+ def generate_string(self):
+ self.string_length += 1
+ return ''.join(random.choices(string.ascii_letters, k=self.string_length))
+
+ def generate_string_array(self):
+ self.array_length += 1
+ return [self.generate_string() for _ in range(self.array_length)]
+
+ def generate_string_list_2d(self, num_rows):
+ return [[self.generate_string() for _ in range(self.array_length)] for _ in range(num_rows)]
+
+ def generate_string_list_2d(self, num_rows, num_cols):
+ return [[self.generate_string() for _ in range(num_cols)] for _ in range(num_rows)]
+
+
+class RandomGeneratorRandom:
+ """
+ One of the supported generators with size of object random in each call to generator functions
+
+ Attributes
+ ----------
+ int_counter : int
+ counter for generating int values
+ string_length : int
+ length of the string to be generated
+ array_length : int
+ length of the array to be generated
+
+ Methods
+ -------
+ generate_int()
+ generates int value
+ generate_string()
+ generates string value of random length
+ generate_array()
+ generates array of random length
+ generate_string_list_2d(num_rows)
+ generates 2d array of strings of maximum size num_rows x random length
+ generate_string_list_2d(num_rows, num_cols)
+ generates 2d array of strings of maximum size num_rows x num_cols
+ """
+ def generate_int(self, max_int=100000000):
+ return int(random.randint(1, max_int))
+
+ def generate_string(self):
+ return ''.join(random.choices(string.ascii_letters, k=self.generate_int(50)))
+
+ def generate_string_array(self):
+ return [self.generate_string() for _ in range(self.generate_int(50))]
+
+ def generate_string_list_2d(self, num_rows):
+ num_rows = random.randint(1, num_rows)
+ array_length = self.generate_int(50)
+ return [[self.generate_string() for _ in range(array_length)] for _ in range(num_rows)]
+
+ def generate_string_list_2d(self, num_rows, num_cols):
+ num_rows = random.randint(1, num_rows)
+ num_cols = random.randint(1, num_cols)
+ return [[self.generate_string() for _ in range(num_cols)] for _ in range(num_rows)]
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/service_object_stores.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/service_object_stores.py
new file mode 100644
index 000000000..2264a392c
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_object_stores/service_object_stores.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from abc import ABC, abstractmethod
+
+from ranger_performance_tool.ranger_perf_object_stores import random_generators
+
+
+class ServiceStoreBase(ABC):
+
+ @abstractmethod
+ def generate_resources(self):
+ pass
+
+
+class HDFSServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ paths = self.random_generator.generate_string_array()
+ resources = {'path': {'values': paths, 'isExcludes': False, 'isRecursive': True}}
+ return resources
+
+
+class HBaseServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ columns = self.random_generator.generate_string_array()
+ tables = self.random_generator.generate_string_array()
+ column_families = self.random_generator.generate_string_array()
+ resources = {'column-family': {'values': column_families, 'isExcludes': False, 'isRecursive': False},
+ 'column': {'values': columns, 'isExcludes': False, 'isRecursive': False},
+ 'table': {'values': tables, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class HiveServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ tables = self.random_generator.generate_string_array()
+ databases = self.random_generator.generate_string_array()
+ columns = self.random_generator.generate_string_array()
+ resources = {'table': {'values': tables, 'isExcludes': False, 'isRecursive': False},
+ 'database': {'values': databases, 'isExcludes': False, 'isRecursive': False},
+ 'column': {'values': columns, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class YarnServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ queues = self.random_generator.generate_string_array()
+ resources = {'queue': {'values': queues, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class KnoxServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ services = self.random_generator.generate_string_array()
+ topologies = self.random_generator.generate_string_array()
+ resources = {'topology': {'values': topologies, 'isExcludes': False, 'isRecursive': False},
+ 'service': {'values': services, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class SolrServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ collections = self.random_generator.generate_string_array()
+ resources = {'collection': {'values': collections, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class KafkaServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ consumer_groups = self.random_generator.generate_string_array()
+ resources = {'consumergroup': {'values': consumer_groups, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class AtlasServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ entity_types = self.random_generator.generate_string_array()
+ entity_classifications = self.random_generator.generate_string_array()
+ entity_business_metadatas = self.random_generator.generate_string_array()
+ entities = self.random_generator.generate_string_array()
+ resources = {'entity-type': {'values': entity_types, 'isExcludes': False, 'isRecursive': False},
+ 'entity-classification': {'values': entity_classifications, 'isExcludes': False,
+ 'isRecursive': False},
+ 'entity': {'values': entities, 'isExcludes': False, 'isRecursive': False},
+ 'entity-business-metadata': {'values': entity_business_metadatas, 'isExcludes': False,
+ 'isRecursive': False}}
+ return resources
+
+
+class OzoneServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ buckets = self.random_generator.generate_string_array()
+ keys = self.random_generator.generate_string_array()
+ volume = self.random_generator.generate_string_array()
+ resources = {'bucket': {'values': buckets, 'isExcludes': False, 'isRecursive': False},
+ 'key': {'values': keys, 'isExcludes': False, 'isRecursive': False},
+ 'volume': {'values': volume, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class AdlsServiceStore(ServiceStoreBase):
+
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ containers = self.random_generator.generate_string_array()
+ paths = self.random_generator.generate_string_array()
+ storage_accounts = self.random_generator.generate_string_array()
+ resources = {'container': {'values': containers, 'isExcludes': False, 'isRecursive': False},
+ 'relativepath': {'values': paths, 'isExcludes': False, 'isRecursive': True},
+ 'storageaccount': {'values': storage_accounts, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class S3ServiceStore(ServiceStoreBase):
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ buckets = self.random_generator.generate_string_array()
+ paths = self.random_generator.generate_string_array()
+ resources = {'bucket': {'values': buckets, 'isExcludes': False, 'isRecursive': False},
+ 'path': {'values': paths, 'isExcludes': False, 'isRecursive': False}}
+ return resources
+
+
+class KuduServiceStore(ServiceStoreBase):
+ def __init__(self, random_type="random"):
+ self.random_generator = random_generators.get_random_generator(random_type)
+
+ def generate_resources(self):
+ columns = self.random_generator.generate_string_array()
+ databases = self.random_generator.generate_string_array()
+ tables = self.random_generator.generate_string_array()
+ resources = {'table': {'values': tables, 'isExcludes': False, 'isRecursive': False},
+ 'database': {'values': databases, 'isExcludes': False, 'isRecursive': False},
+ 'column': {'values': columns, 'isExcludes': False, 'isRecursive': False}}
+ return resources
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/__init__.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/config_utils.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/config_utils.py
new file mode 100644
index 000000000..1641b353e
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/config_utils.py
@@ -0,0 +1,85 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import json
+
+
+class ConfigReader:
+ """
+ Reads the config files and supports requested values from the user
+ Methods:
+ get_config_value: returns the value of the config key
+ override_with_command_line_args: overrides the config values with command line arguments
+ """
+ def __init__(self, primary_config_file, secondary_config_file):
+ """
+ Constructor
+ :param primary_config_file: str , path to primary config file
+ :param secondary_config_file: str , path to secondary config file
+ """
+ try:
+ with open(primary_config_file, 'r') as f:
+ self.primary_config = json.load(f)
+ except Exception as e:
+ raise Exception("Error reading primary config file: {}. Run reset.py script to reinitialize configs".format(primary_config_file))
+ try:
+ with open(secondary_config_file, 'r') as f:
+ self.secondary_config = json.load(f)
+ except Exception as e:
+ raise Exception("Error reading secondary config file: {}. Run reset.py script to reinitialize configs".format(secondary_config_file))
+
+ def override_with_command_line_args(self, arg_dict):
+ """
+ Overrides the config values with command line arguments
+ :param arg_dict: dict , command line arguments
+ :return: void
+ """
+ api_name = arg_dict["api"]
+ self.primary_config["ranger_url"] = arg_dict["ranger_url"]
+ self.primary_config["api_list"] = [api_name]
+ self.primary_config["api"][api_name]["num_calls"] = int(arg_dict["calls"])
+ self.primary_config["system_logger"]["num_calls"] = int(arg_dict["calls"]) + 5
+ self.primary_config["ranger_auth"] = (arg_dict["username"], arg_dict["password"])
+ self.primary_config["host_name"] = arg_dict["ssh_host"]
+ self.primary_config["user"] = arg_dict["ssh_user"]
+ self.primary_config["password"] = arg_dict["ssh_password"]
+ self.primary_config["client_ip"] = arg_dict["client_ip"]
+
+ def get_config_value(self, config_file="primary", *key_args):
+ """
+ Returns the value of the config key
+ :param config_file: str , config file type - primary or secondary
+ :param key_args: list , list of keys to get the value of. Add multiple to dtech nested config values.
+ :return: config_value: str , value corresponding to the config key
+ """
+ if len(key_args) == 0:
+ raise Exception("No config key specified")
+ if len(key_args) > 3:
+ raise Exception("Too many levels of nesting requested. Maximum of 3 levels allowed.")
+ if config_file == "primary":
+ config_dict = self.primary_config
+ elif config_file == "secondary":
+ config_dict = self.secondary_config
+ else:
+ raise Exception("Invalid config file name. Must be either primary or secondary.")
+
+ if len(key_args) == 1:
+ return config_dict[key_args[0]]
+ elif len(key_args) == 2:
+ return config_dict[key_args[0]][key_args[1]]
+ elif len(key_args) == 3:
+ return config_dict[key_args[0]][key_args[1]][key_args[2]]
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/dataframe_utils.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/dataframe_utils.py
new file mode 100644
index 000000000..01e96a798
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/dataframe_utils.py
@@ -0,0 +1,129 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pandas as pd
+
+
+class DataframeUtils:
+ """
+ Utilities for dataframe manipulation
+ Methods
+ ----------
+ truncate_dataframe(data, start_time, end_time, timestamp_col_name='time')
+ truncates dataframe to given time range
+ align_dataframes(system_logs_df, access_logs_df, system_logs_timestamp_col_name = 'time',
+ access_logs_timestamp_col_name = 'time', timestamp_truncation=True, merge=False)
+ aligns two dataframes based on timestamp column name
+ combine_system_logs_dataframe(main_system_logs_df, secondary_system_logs_df)
+ combines two system logs dataframes
+ insert_column(dataframe, column_name, column_data)
+ inserts column to dataframe
+ rename_rows(dataframe, row_name_mapping)
+ renames rows in dataframe
+ rename_columns(dataframe, column_name_mapping)
+ renames columns in dataframe
+ """
+ def __init__(self):
+ pass
+
+ def truncate_dataframe(self, data, start_time, end_time, timestamp_col_name='time'):
+ """
+ truncates dataframe to given time range
+ :param data: Pandas dataframe to be truncated
+ :param start_time: First time to be included in the dataframe
+ :param end_time: Last time to be included in the dataframe
+ :param timestamp_col_name: Column name of the timestamp column
+ :return: Pandas dataframe with truncated data
+ """
+ mask = (pd.to_datetime(data[timestamp_col_name]) >= pd.to_datetime(start_time, infer_datetime_format=True)) & (
+ pd.to_datetime(data[timestamp_col_name]) <= pd.to_datetime(end_time, infer_datetime_format=True))
+ return data.loc[mask]
+
+ def align_dataframes(self, system_logs_df, access_logs_df, system_logs_timestamp_col_name = 'time',
+ access_logs_timestamp_col_name = 'time', timestamp_truncation=True, merge=False):
+ """
+ aligns two dataframes based on timestamp column name
+ :param system_logs_df: Pandas dataframe of system logs
+ :param access_logs_df: Pandas dataframe of access logs
+ :param system_logs_timestamp_col_name: Name of the timestamp column in system logs dataframe
+ :param access_logs_timestamp_col_name: Name of the timestamp column in access logs dataframe
+ :param timestamp_truncation: bool, True if timestamp truncation is to be done
+ :param merge: bool, True if dataframes are to be merged
+ :return: Pandas dataframe. Based on merge and timestamp truncation parameters
+ """
+ if system_logs_timestamp_col_name != 'time':
+ system_logs_df.rename(columns={system_logs_timestamp_col_name: 'time'}, inplace=True)
+ system_logs_timestamp_col_name = 'time'
+
+ if access_logs_timestamp_col_name != 'time':
+ access_logs_df.rename(columns={access_logs_timestamp_col_name: 'time'}, inplace=True)
+ access_logs_timestamp_col_name = 'time'
+
+ if timestamp_truncation:
+ start_time = system_logs_df[system_logs_timestamp_col_name].min()
+ end_time = system_logs_df[system_logs_timestamp_col_name].max()
+ system_logs_df = self.truncate_dataframe(system_logs_df, start_time, end_time,
+ timestamp_col_name=system_logs_timestamp_col_name)
+ access_logs_df = self.truncate_dataframe(access_logs_df, start_time, end_time,
+ timestamp_col_name=access_logs_timestamp_col_name)
+ if merge:
+ aligned_df = pd.merge(access_logs_df, system_logs_df, on=access_logs_timestamp_col_name, how='outer', sort=True)
+ else:
+ aligned_df = access_logs_df
+ return aligned_df
+
+ def combine_system_logs_dataframe(self, main_system_logs_df, secondary_system_logs_df):
+ """
+ Concats two system logs dataframes
+ :param main_system_logs_df: Pandas dataframe of system logs
+ :param secondary_system_logs_df: Pandas dataframe of secondary system logs
+ :return: Pandas dataframe of combined system logs
+ """
+ secondary_system_logs_df = pd.concat([secondary_system_logs_df]*main_system_logs_df.shape[0], ignore_index=True)
+ combined_df = pd.concat([main_system_logs_df, secondary_system_logs_df], axis=1)
+ return combined_df
+
+ def insert_column(self, dataframe, column_name, column_data):
+ """
+ Inserts column to dataframe
+ :param dataframe: Pandas dataframe to be modified
+ :param column_name: Name of the column to be inserted
+ :param column_data: Data to be inserted in the column
+ :return: Pandas dataframe with inserted column
+ """
+ last_index = len(dataframe.columns)
+ dataframe.insert(last_index, column_name, column_data)
+ return dataframe
+
+ def rename_rows(self, dataframe, row_name_mapping):
+ """
+ Renames rows in dataframe
+ :param dataframe: Pandas dataframe to be modified
+ :param row_name_mapping: dict, mapping of old row names to new row names
+ :return: void, modifies dataframe in place
+ """
+ dataframe.rename(index=row_name_mapping, inplace=True)
+
+ def rename_columns(self, dataframe, column_name_mapping):
+ """
+ Renames columns in dataframe
+ :param dataframe: Pandas dataframe to be modified
+ :param column_name_mapping: dict, mapping of old column names to new column names
+ :return: void, modifies dataframe in place
+ """
+ dataframe.rename(columns=column_name_mapping, inplace=True)
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/extra_unused_utils.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/extra_unused_utils.py
new file mode 100644
index 000000000..5545dd8c5
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/extra_unused_utils.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# #contains some of the code/features that were developed initially but were removed from the main codebase
+#
+# from tinydb import TinyDB, Query
+#
+# from ranger_performance_tool.ranger_perf_utils.logging_utils import LogParser
+#
+#
+# class DatabaseStore:
+#
+# def __init__(self, db_type='tiny_db', db_file='/tmp/ranger_tiny_db.json'):
+# if db_type == 'tiny_db':
+# self.db = TinyDBStore(db_file)
+# else:
+# raise Exception('Unknown database type: {}'.format(db_type))
+#
+# def insert(self, obj):
+# return self.db.insert(obj)
+#
+# def get_all(self):
+# return self.db.get_all()
+#
+# def get_one(self):
+# return self.db.get_one()
+#
+# def delete_one(self):
+# return self.db.delete_one()
+#
+# def delete_all(self):
+# return self.db.delete_all()
+#
+#
+# class TinyDBStore:
+#
+# def __init__(self, db_file):
+# self.db = TinyDB(db_file)
+#
+# def insert(self, obj):
+# self.db.insert(obj)
+#
+# def get_all(self):
+# return self.db.all()
+#
+# def get_one(self):
+# all = self.db.all()
+# if len(all) == 0:
+# raise Exception('No elements in database for get_one. Please insert an element first')
+# return all[0]
+#
+# def delete_one(self):
+# all = self.db.all()
+# if len(all) == 0:
+# raise Exception('No elements in database for delete_one. Please insert an element first')
+# ele = all[0]
+# self.db.remove(doc_ids=[ele.doc_id])
+# return ele
+#
+# def delete_all(self):
+# self.db.truncate()
+#
+# class DataframeMetrics:
+#
+# def get_top_threshold(self, dataframe, column, percentile):
+# threshold = dataframe[column].quantile(percentile / 100)
+# return threshold
+#
+# def get_bottom_threshold(self, dataframe, column, percentile):
+# threshold = dataframe[column].quantile(1 - percentile / 100)
+# return threshold
+#
+#
+# def add_additional_statistic(self, dataframe, statistic_name, to_remove_column_name=None):
+# try:
+# if statistic_name == 'memory_used_percentage':
+# column_name = LogParser().get_header_mapping_system_logs('free')
+# dataframe['memory_used_percentage'] = (dataframe['total_memory'] - dataframe[column_name])/dataframe['total_memory']
+# else:
+# raise Exception(f"Unimplemented statistic: {statistic_name}")
+#
+# except:
+# Exception(f"Error while adding statistic {statistic_name}")
+# return dataframe
\ No newline at end of file
diff --git a/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/logging_utils.py b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/logging_utils.py
new file mode 100644
index 000000000..a6dd7ecf7
--- /dev/null
+++ b/ranger-tools/src/main/python/ranger_performance_tool/ranger_perf_utils/logging_utils.py
@@ -0,0 +1,293 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from datetime import datetime
+from collections import defaultdict
+
+import pandas as pd
+
+
+class LogFetcher:
+ """
+ Copies system logs and access logs from server to local machine
+
+ Methods:
+ ----------
+ fetch_system_logs_from_server(host, username, password, remote_file,
+ local_path="/tmp/") : fetches system logs from server and copies to local machine
+ fetch_secondary_system_logs_from_server(host, username, password, remote_file,
+ local_path="/tmp/") : fetches secondary system logs from server and copies to local machine
+ fetch_access_logs_from_server(host, username, password, remote_path,
+ local_path="/tmp/") : fetches access logs from server and copies to local machine
+ """
+ def fetch_system_logs_from_server(self, host, username, password, remote_file,
+ local_path="/tmp/"):
+ """
+ Fetches system logs from server and copies to local machine
+ :param host: str, ssh host
+ :param username: str, ssh username
+ :param password: str, ssh password
+ :param remote_file: str, location of system logs on server
+ :param local_path: str, local path to copy system logs to
+ :return: local_file: str, local file path to system logs
+ """
+ local_file = f"{local_path}system_logs.txt"
+ command = f"sshpass -p {password} scp -o StrictHostKeyChecking=no {username}@{host}:{remote_file} {local_file}"
+ print(f"fetch command = {command}")
+ os.system(command)
+ return local_file
+
+ def fetch_secondary_system_logs_from_server(
+ self, host, username, password, remote_file, local_path="/tmp/"):
+ """
+ Fetches secondary system logs from server and copies to local machine
+ :param host: str, ssh host
+ :param username: str, ssh username
+ :param password: str, ssh password
+ :param remote_file: str, location of system logs on server
+ :param local_path: str, local path to copy system logs to
+ :return: local_file: str, local file path to system logs
+ """
+ local_file = f"{local_path}secondary_system_logs.txt"
+ command = f"sshpass -p {password} scp -o StrictHostKeyChecking=no {username}@{host}:{remote_file} {local_file}"
+ print(f"fetch command = {command}")
+ os.system(command)
+ return local_file
+
+ def fetch_access_logs_from_server(self, host, username, password, remote_path, local_path="/tmp/"):
+ """
+ Fetches access logs from server and copies to local machine
+ :param host: str, ssh host
+ :param username: str, ssh username
+ :param password: str, ssh password
+ :param local_path: str, local path to copy access logs to
+ :param remote_path: str, remote path to copy access logs from
+ :return: local_file: str, local file path to access logs
+ """
+ local_file = f"{local_path}access_logs.txt"
+ remote_file = f"{remote_path}access_log-{host}-{datetime.utcnow().strftime('%Y-%m-%d')}.log"
+ command = f"sshpass -p {password} scp -o StrictHostKeyChecking=no {username}@{host}:{remote_file} {local_file}"
+ print(f"fetch command = {command}")
+ os.system(command)
+ return local_file
+
+
+class LogParser:
+ """
+ Parse access logs and system logs
+ Attributes:
+ header_mapping_system_logs: dict, mapping of system logs header to column name in dataframe
+
+ Methods:
+ get_header_mapping_system_logs: returns dict of system logs header to column name in dataframe
+ parse_secondary_system_logs: parse secondary system logs and return a dataframe
+ parse_system_logs: parse system logs and return a dataframe
+
+ """
+ header_mapping_system_logs = {
+ 'r': 'Active process count',
+ 'b': 'Sleeping process count',
+ 'swpd': 'Total virtual memory swap (MegaBytes)',
+ 'free': 'Total free memory (MegaBytes)',
+ 'buff': 'Total memory temporarily used as a data buffer (MegaBytes)',
+ 'cache': 'Total cache memory (MegaBytes)',
+ 'si': 'The rate of swapping-in memory from disk',
+ 'so': 'The rate of swapping-out memory to disk',
+ 'bi': 'Blocks received from a block device per second',
+ 'bo': 'Blocks sent to a block device per second',
+ 'in': 'The number of system interrupts',
+ 'cs': 'The number of context switches per second',
+ 'us': 'The percentage of CPU time spent on non-kernel processes',
+ 'sy': 'The percentage of CPU time spent on kernel processes',
+ 'id': 'The percentage of idle CPU',
+ 'wa': 'The percentage of CPU time spent waiting for Input/Output',
+ 'st': 'The percentage of CPU time stolen by a virtual machine',
+ 'UTC': 'time'
+ }
+
+ @staticmethod
+ def get_header_mapping_system_logs(key):
+ """
+ Get the header mapping for system logs
+ :param key: column name from vmstat command
+ :return: header mapping for system logs for given column name
+ """
+ return LogParser.header_mapping_system_logs[key]
+
+ @staticmethod
+ def _parse_string(x):
+ return x[1:-1]
+
+ @staticmethod
+ def _parse_datetime(x):
+ dt = datetime.strptime(x[1:-7], '%d/%b/%Y:%H:%M:%S')
+ return dt
+
+ def _combine_date_time(self, line_split):
+ line_split[-2] = line_split[-2] + " " + line_split[-1]
+ del line_split[-1]
+
+ def parse_secondary_system_logs(self, local_file):
+ """
+ Parse system logs and return a dataframe with the following columns: total_memory.
+ Can be extended to other columns if needed.
+ :param local_file: path to the secondary system logs file on the local machine
+ :return: dataframe of system logs
+ """
+ print("Ensure command executed for creating secondary system logs is similar in schema produced to :: "
+ "vmstat -s -S M")
+ secondary_sys_logs = {}
+ with open(local_file, 'r') as f:
+ lines = f.readlines()
+ total_memory = lines[0].split()[0]
+ secondary_sys_logs['total_memory (MegaBytes)'] = [total_memory]
+
+ secondary_df = pd.DataFrame(secondary_sys_logs)
+ return secondary_df
+
+ # skipping first line which is the header:: procs --memory---swap---io- -system--cpu----timestamp
+ # Second line is header for the rest of the lines:: r/b/swpd/free/buff/cache/si/so/bi/bo/in/cs/us/sy/id/wa/st/UTC
+ def parse_system_logs(self, local_file, metrics=None):
+ """
+ Parse system logs and return a dataframe with the following columns:
+ :param local_file: path to the system logs file on the local machine
+ :param metrics: list of metrics to be parsed from the system logs
+ :return: dataframe with columns as specified in metrics
+ """
+ print("Ensure command executed for creating system logs is similar in schema produced to :: vmstat 5 -S M -n -t")
+ with open(local_file, 'r') as f:
+ lines = f.readlines()[1:]
+ headers = lines[0].split()
+ print("len headers= ", len(headers))
+ dict_lists = defaultdict(list)
+ print(f"header = {headers}")
+ for line in lines[1:]:
+ line_split = line.split()
+ # assume last 2 columns (after splitting) are date and time
+ self._combine_date_time(line_split)
+ for i,val in enumerate(line_split):
+ dict_lists[headers[i]].append(val)
+ dict_lists['UTC'] = pd.to_datetime(dict_lists['UTC'], infer_datetime_format=True)
+ if metrics is not None:
+ dict_lists = {key: dict_lists[key] if key in headers else Exception(f"Requested metric '{key}' not found in system logs")
+ for key in metrics}
+ df = pd.DataFrame(dict_lists)
+ cols = df.columns
+ for c in cols:
+ if c != 'UTC':
+ try:
+ df[c] = pd.to_numeric(df[c])
+ except:
+ pass
+ return df
+
+ def parse_access_logs(self, local_file, ip_address_client):
+ """
+ Parse access logs and return a dataframe. Currently only supported for regular expressions related to policy API's
+ :param local_file: path to access logs file
+ :param ip_address_client: ip address of client which is used to filter access logs
+ :return: dataframe of access logs
+ """
+ print("Ensure ranger.accesslog.pattern to include the %D in the access pattern so that the tomcat server "
+ "also logs the api execution time")
+ data = pd.read_csv(
+ local_file,
+ sep=r'\s(?=(?:[^"]*"[^"]*")*[^"]*$)(?![^\[]*\])',
+ engine='python',
+ na_values='-',
+ header=None,
+ usecols=[0, 3, 4, 9],
+ names=['ip', 'time', 'request', 'latency'],
+ converters={'time': LogParser._parse_datetime,
+ 'request': LogParser._parse_string})
+
+ data = data[data['ip'] == ip_address_client]
+ data = data[data['request'].str.match('.*service/public/v2/api/policy/[0-9]*') | data['request'].str.match(
+ '^POST.*service/public/v2/api/policy')]
+ data['time'] = pd.to_datetime(data['time'], format="%d/%b/%Y:%H:%M:%S")
+ data[['type', 'url', 'http']] = data.request.str.split(' ', expand=True)
+ data = data.drop(columns=['request', 'http'])
+ return data
+
+
+class SystemLogger:
+ """
+ Class to log system metrics
+
+ Methods:
+ delete_old_logs() - deletes old system logs
+ start_system_log_service() - starts system log service on the server using vmstat command
+ execute_secondary_system_log_command() - executes secondary system log command once on the server using vmstat command with -s which gives detailed system metrics
+ stop_system_log_service() - stops system log service started by start_system_log_service() on the server
+ """
+ # TODO: change os.system to subprocess.call
+ def __init__(self, host, username, password, log_file, secondary_log_file):
+ """
+ Initializes the SystemLogger class
+ :param host: ssh host
+ :param username: ssh username of server
+ :param password: ssh password
+ :param log_file: file path where to store system logs
+ :param secondary_log_file: file path where to store secondary system logs
+ """
+ self.log_file = log_file
+ self.secondary_log_file = secondary_log_file
+ self.command_prefix = f"sshpass -p {password} ssh -o StrictHostKeyChecking=no {username}@{host}"
+
+ def delete_old_logs(self):
+ """
+ Deletes old system logs
+ :return: None
+ """
+ command = f"{self.command_prefix} yes | rm {self.log_file}"
+ os.system(command)
+
+ def start_system_log_service(self, sleep_seconds, num_entries):
+ """
+ Starts system log service on the server using vmstat command
+ :param sleep_seconds: time gap between system log entries
+ :param num_entries: total number of system log entries to be stored
+ :return: None
+ """
+ try:
+ num_entries = int(num_entries)
+ command = f"{self.command_prefix} 'vmstat {sleep_seconds} {num_entries} -S M -n -t > {self.log_file}' &"
+ except Exception as e:
+ print("num_entries must be an integer --- executing vmstat for infinite time")
+ command = f"{self.command_prefix} 'vmstat {sleep_seconds} -S M -n -t > {self.log_file}' &"
+ print(f"command = {command}")
+ os.system(command)
+
+ def execute_secondary_system_log_command(self):
+ """
+ Executes secondary system log command once on the server using vmstat command with -s which gives detailed system metrics
+ :return: None
+ """
+ command = f"{self.command_prefix} 'vmstat -s -S M > {self.secondary_log_file}' &"
+ print(f"command = {command}")
+ os.system(command)
+
+ def stop_system_log_service(self):
+ """
+ Stops system log service started by start_system_log_service() on the server
+ :return: None
+ """
+ command = f"{self.command_prefix} pkill -f 'vmstat'"
+ print(f"command = {command}")
+ os.system(command)
diff --git a/ranger-tools/src/main/python/requirements.txt b/ranger-tools/src/main/python/requirements.txt
new file mode 100644
index 000000000..1a83e6e07
--- /dev/null
+++ b/ranger-tools/src/main/python/requirements.txt
@@ -0,0 +1,22 @@
+apache-ranger==0.0.6
+certifi==2022.6.15
+charset-normalizer==2.1.0
+cycler==0.11.0
+fonttools==4.34.4
+idna==3.3
+Jinja2==3.1.2
+kiwisolver==1.4.4
+MarkupSafe==2.1.1
+matplotlib==3.5.3
+numpy==1.23.1
+packaging==21.3
+pandas==1.4.3
+Pillow==9.2.0
+pyparsing==3.0.9
+python-dateutil==2.8.2
+pytz==2022.2
+requests==2.28.1
+scipy==1.9.0
+seaborn==0.11.2
+six==1.16.0
+urllib3==1.26.11
diff --git a/ranger-tools/src/main/python/setup_performance_analyzer.py b/ranger-tools/src/main/python/setup_performance_analyzer.py
new file mode 100644
index 000000000..687fc4776
--- /dev/null
+++ b/ranger-tools/src/main/python/setup_performance_analyzer.py
@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Use this script to reinitialize config files for performance analyser. Call this only once for a particular client machine.
+Running this script will override your existing config files. Take back up your existing config files before running this script.
+
+usage: python3 setup_performance_analyzer.py
+"""
+
+import os
+import json
+
+from ranger_performance_tool.ranger_perf_assets.path import DEFAULT_PRIMARY_CONFIG_PATH, DEFAULT_SECONDARY_CONFIG_PATH
+
+CONFIG_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), 'config/'))
+
+to_write_primary_config = os.path.join(CONFIG_DIR, 'primary_config.json')
+to_write_secondary_config = os.path.join(CONFIG_DIR, 'secondary_config.json')
+
+
+def overwrite_config(config_file, config_dict):
+ with open(config_file, 'w') as f:
+ json.dump(config_dict, f, indent=4)
+
+
+def read_config(config_file):
+ with open(config_file, 'r') as f:
+ return json.load(f)
+
+
+print("Reading default primary config file from {}".format(DEFAULT_PRIMARY_CONFIG_PATH))
+print("Reading default secondary config file from {}".format(DEFAULT_SECONDARY_CONFIG_PATH))
+print("Writing to primary config path: {}".format(to_write_primary_config))
+print("Writing to secondary config path: {}".format(to_write_secondary_config))
+
+default_primary_json = read_config(DEFAULT_PRIMARY_CONFIG_PATH)
+overwrite_config(to_write_primary_config, default_primary_json)
+
+default_secondary_json = read_config(DEFAULT_SECONDARY_CONFIG_PATH)
+overwrite_config(to_write_secondary_config, default_secondary_json)
\ No newline at end of file