You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ranger.apache.org by "ChinmayHegde24 (via GitHub)" <gi...@apache.org> on 2025/03/24 11:28:10 UTC

[PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

ChinmayHegde24 opened a new pull request, #547:
URL: https://github.com/apache/ranger/pull/547

   This PR involves developing a functional test suite using Pytest to validate the Key Management Service (KMS) REST API endpoints and test HDFS encryption functionalities. The scope includes:
   
   Automating API endpoint tests for key creation, deletion and further operation on keys.
   Managing encryption zones in HDFS and validating key association.
   Performing read/write operations on encrypted files.
   Verifying access control and permissions within encrypted zones.
   
   For further information please refer  [PyTest-KMS-HDFS/readme.md]
                                                                [PyTest-KMS-HDFS/test_hdfs/readme.md]
                                                                [PyTest-KMS-HDFS/test_kms/readme.md]
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141386348


##########
PyTest-KMS-HDFS/test_kms/test_keyDetails.py:
##########
@@ -0,0 +1,168 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+import requests

Review Comment:
   Avoid using `requests` directly, see https://pypi.org/project/apache-ranger/ for examples of how to leverage `RangerKMSClient` in `ranger_kms_client.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on PR #547:
URL: https://github.com/apache/ranger/pull/547#issuecomment-2964714059

   Is it required for the directories to be named with `test_` in the prefix? like `test_hdfs`, `test_kms`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141389047


##########
PyTest-KMS-HDFS/test_kms/test_keyOps.py:
##########
@@ -0,0 +1,239 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+
+import requests

Review Comment:
   Avoid using `requests` directly, see https://pypi.org/project/apache-ranger/ for examples of how to leverage `RangerKMSClient` in `ranger_kms_client.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on PR #547:
URL: https://github.com/apache/ranger/pull/547#issuecomment-2964716641

   I suggest keeping a generic name for test directory instead of `PyTest-KMS-HDFS`, the test suite can be extended later to add more services.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141374222


##########
PyTest-KMS-HDFS/test_kms/conftest.py:
##########
@@ -0,0 +1,60 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+import pytest
+import requests

Review Comment:
   Avoid using `requests` directly, see https://pypi.org/project/apache-ranger/ for examples of how to leverage `RangerKMSClient` in `ranger_kms_client.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "ChinmayHegde24 (via GitHub)" <gi...@apache.org>.

ChinmayHegde24 closed pull request #547: RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption
URL: https://github.com/apache/ranger/pull/547


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141364127


##########
PyTest-KMS-HDFS/readme.md:
##########
@@ -0,0 +1,94 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+
+#    KMS API & HDFS Encryption Pytest Suite
+
+
+This test suite validates REST API endpoints for KMS (Key Management Service) and tests HDFS encryption functionalities including key management and file operations within encryption zones.
+
+**test_kms  :** contains test cases for checking KMS API functionality  
+
+**test_hdfs :** contains test cases for checking KMS functionality through hdfs encryption lifecycle
+
+## 📂 Directory Structure
+
+```
+test_directory/
+├── test_kms/                # Tests on KMS API

Review Comment:
   Suggested directory structure:
   
       tests
       ├──hdfs
           ├── test_encryption.py
           ├── test_audit_spooling.py  
           ├── test_config.py         
           ├── conf-test.py           
           ├── readme.md
           ├── utils.py 
       ├──kms  
           ├── test_keys.py                  
           ├── test_blacklisting.py  
           ├── conf-test.py
           ├── utils.py 
           ├── readme.md



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141390718


##########
PyTest-KMS-HDFS/test_kms/test_keys.py:
##########
@@ -0,0 +1,100 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+import requests

Review Comment:
   Avoid using `requests` directly, see https://pypi.org/project/apache-ranger/ for examples of how to leverage `RangerKMSClient` in `ranger_kms_client.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "pradeepagrawal8184 (via GitHub)" <gi...@apache.org>.

pradeepagrawal8184 commented on PR #547:
URL: https://github.com/apache/ranger/pull/547#issuecomment-3877359376

   The PR has 56 commits reduce it to 1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141379885


##########
PyTest-KMS-HDFS/test_kms/test_blacklisting.py:
##########
@@ -0,0 +1,316 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+
+import xml.etree.ElementTree as ET
+import requests
+import pytest
+import time
+import docker
+import tarfile
+import io
+
+
+DBKS_SITE_PATH = "/opt/ranger/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml"
+KMS_SERVICE_NAME = "dev_kms"
+BASE_URL = "http://localhost:9292/kms/v1"   
+RANGER_AUTH = ('keyadmin', 'rangerR0cks!')  # Ranger key admin user
+BASE_URL_RANGER = "http://localhost:6080/service/public/v2/api/policy"
+HEADERS={"Content-Type": "application/json","Accept":"application/json"}
+KMS_CONTAINER_NAME = "ranger-kms"
+
+test_user="nobody"
+
+client = docker.from_env()
+container = client.containers.get(KMS_CONTAINER_NAME)
+
+
+@pytest.fixture(scope="module", autouse=True)
+def create_test_user():
+    # Add user inside the container
+    exit_code, output = container.exec_run(f"useradd {test_user}", user="root")
+
+    # Proceed if user added successfully or already exists
+    if exit_code != 0 and "already exists" not in output.decode():
+        raise Exception(f"Failed to create user: {output.decode()}")
+
+    print(f"✅ Created test user: {test_user}")
+
+    yield test_user
+
+    # Clean up the user after test
+    exit_code, output = container.exec_run(f"userdel {test_user}", user="root")
+    if exit_code == 0:
+        print(f"🧹 Deleted test user: {test_user}")
+    else:
+        print(f"⚠️ Warning: Failed to delete test user: {output.decode()}")
+
+
+@pytest.fixture(scope="module")
+def user1(create_test_user):
+    return create_test_user
+
+
+# **************** create KMS policy for user1 --------------------------------------
+@pytest.fixture(scope="module", autouse=True)
+def create_initial_kms_policy(user1):
+    policy_data = {
+        "policyName": "blacklist-policy",
+        "service": KMS_SERVICE_NAME,
+        "resources": {
+            "keyname": {
+                "values": ["blacklist-*"],  # Match any key starting with 'blacklist-'
+                "isExcludes": False,
+                "isRecursive": False
+            }
+        },
+        "policyItems": [{
+            "accesses": [
+                {"type": "CREATE", "isAllowed": True},
+                {"type": "ROLLOVER", "isAllowed": True},
+                {"type": "DELETE", "isAllowed": True}
+            ],
+            "users": [user1]
+        }]
+    }
+
+    response = requests.post(BASE_URL_RANGER, auth=RANGER_AUTH, json=policy_data)

Review Comment:
   Avoid using `requests` directly, see https://pypi.org/project/apache-ranger/ for examples of how to leverage `RangerClient` in `ranger_client.py`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "ChinmayHegde24 (via GitHub)" <gi...@apache.org>.

ChinmayHegde24 commented on PR #547:
URL: https://github.com/apache/ranger/pull/547#issuecomment-2965203527

   > Is it required for the directories to be named with `test_` in the prefix? like `test_hdfs`, `test_kms`?
   
   No, It's not necessary 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.

Copilot commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2114809295


##########
PyTest-KMS-HDFS/test_kms/utils.py:
##########
@@ -0,0 +1,14 @@
+#no scope mismatch due to utils

Review Comment:
   [nitpick] This comment is unclear—consider clarifying the intent or removing it if it isn't adding meaningful context.



##########
test-pytest.sh:
##########
@@ -0,0 +1,110 @@
+#!/bin/bash
+
+#handle input
+if [ $# -lt 1 ]; then
+  echo "no arguments passed , using default DB: postgres"
+  DB_TYPE=postgres
+  EXTRA_SERVICES=()
+
+else
+  DB_TYPE=$1
+  shift
+  EXTRA_SERVICES=("$@")
+fi
+
+# Remove all containers and clean up docker space
+docker rm -f $(docker ps -aq --filter "name=ranger")
+docker system prune --all --force --volumes
+
+#path setup
+RANGER_DOCKER_PATH="dev-support/ranger-docker"
+TESTS_PATH="$HOME/Desktop/PyTest-KMS-EP"

Review Comment:
   The `TESTS_PATH` points to `PyTest-KMS-EP` but the test directory is `PyTest-KMS-HDFS`; update the path to the correct test folder.
   ```suggestion
   TESTS_PATH="$HOME/Desktop/PyTest-KMS-HDFS"
   ```



##########
PyTest-KMS-HDFS/test_kms/test_blacklisting.py:
##########
@@ -0,0 +1,263 @@
+import xml.etree.ElementTree as ET
+import requests
+import pytest
+import time
+import docker
+import tarfile
+import io
+
+DBKS_SITE_PATH = "/opt/ranger/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml"
+KMS_SERVICE_NAME = "dev_kms"
+BASE_URL = "http://localhost:9292/kms/v1"   
+RANGER_AUTH = ('keyadmin', 'rangerR0cks!')  # Ranger key admin user
+BASE_URL_RANGER = "http://localhost:6080/service/public/v2/api/policy"
+HEADERS={"Content-Type": "application/json","Accept":"application/json"}
+KMS_CONTAINER_NAME = "ranger-kms"
+
+user1="nobody"
+
+client = docker.from_env()
+container = client.containers.get(KMS_CONTAINER_NAME)
+
+
+# **************** create KMS policy for user1 --------------------------------------
+@pytest.fixture(scope="module", autouse=True)
+def create_initial_kms_policy():
+    policy_data = {
+        "policyName": "blacklist-policy",
+        "service": KMS_SERVICE_NAME,
+        "resources": {
+            "keyname": {
+                "values": ["blacklist-*"],  # Match any key starting with 'blacklist-'
+                "isExcludes": False,
+                "isRecursive": False
+            }
+        },
+        "policyItems": [{
+            "accesses": [
+                {"type": "CREATE", "isAllowed": True},
+                {"type": "ROLLOVER", "isAllowed": True},
+                {"type": "DELETE", "isAllowed": True}
+            ],
+            "users": [user1]
+        }]
+    }
+
+    response = requests.post(BASE_URL_RANGER, auth=RANGER_AUTH, json=policy_data)
+    time.sleep(30)  # Wait for policy propagation

Review Comment:
   Fixed `sleep` delays can slow down tests and add flakiness; consider polling the service for readiness instead of a blind 30s wait.



##########
PyTest-KMS-HDFS/test_hdfs/test_config.py:
##########
@@ -0,0 +1,79 @@
+
+##Contains all constant values regarding USER, PATH, HDFS Commands----------------------
+
+
+HDFS_USER = "hdfs"
+HIVE_USER = "hive"
+HBASE_USER= "hbase"
+KEY_ADMIN="keyadmin"
+HEADERS={"Content-Type": "application/json","Accept":"application/json"}
+PARAMS={"user.name":"keyadmin"}
+BASE_URL="http://localhost:9292/kms/v1"
+HADOOP_CONTAINER = "ranger-hadoop"
+HDFS_USER = "hdfs"

Review Comment:
   Duplicate definition of `HDFS_USER`; remove the redundant assignment to reduce confusion.
   ```suggestion
   # Line 13 removed as it is redundant.
   ```



##########
PyTest-KMS-HDFS/test_kms/utils.py:
##########
@@ -0,0 +1,14 @@
+#no scope mismatch due to utils
+import subprocess
+
+KMS_CONTAINER_NAME = "ranger-kms"             # Replace with your actual KMS container name
+KMS_LOG_FILE = "/var/log/ranger/kms/ranger-kms-ranger-kms.example.com-root.log"
+
+def fetch_logs():
+    try:
+        cmd = f"docker exec {KMS_CONTAINER_NAME} tail -n 100 {KMS_LOG_FILE}"
+        logs = subprocess.check_output(cmd, shell=True, text=True)
+        error_logs = [line for line in logs.split("\n") if "ERROR" in line or "Exception" in line]
+        return "\n".join(error_logs) if error_logs else "No recent errors in logs."
+    except Exception as e:
+        return f"Failed to fetch logs from container: {str(e)}"

Review Comment:
   [nitpick] Catching all exceptions can mask specific errors; consider catching only the exceptions you expect (e.g., `subprocess.CalledProcessError`).
   ```suggestion
       except subprocess.CalledProcessError as e:
           return f"Failed to fetch logs from container. Command failed with exit code {e.returncode}: {e.output}"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "ChinmayHegde24 (via GitHub)" <gi...@apache.org>.

ChinmayHegde24 commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2115796947


##########
PyTest-KMS-HDFS/test_kms/test_blacklisting.py:
##########
@@ -0,0 +1,263 @@
+import xml.etree.ElementTree as ET
+import requests
+import pytest
+import time
+import docker
+import tarfile
+import io
+
+DBKS_SITE_PATH = "/opt/ranger/ranger-3.0.0-SNAPSHOT-kms/ews/webapp/WEB-INF/classes/conf/dbks-site.xml"
+KMS_SERVICE_NAME = "dev_kms"
+BASE_URL = "http://localhost:9292/kms/v1"   
+RANGER_AUTH = ('keyadmin', 'rangerR0cks!')  # Ranger key admin user
+BASE_URL_RANGER = "http://localhost:6080/service/public/v2/api/policy"
+HEADERS={"Content-Type": "application/json","Accept":"application/json"}
+KMS_CONTAINER_NAME = "ranger-kms"
+
+user1="nobody"
+
+client = docker.from_env()
+container = client.containers.get(KMS_CONTAINER_NAME)
+
+
+# **************** create KMS policy for user1 --------------------------------------
+@pytest.fixture(scope="module", autouse=True)
+def create_initial_kms_policy():
+    policy_data = {
+        "policyName": "blacklist-policy",
+        "service": KMS_SERVICE_NAME,
+        "resources": {
+            "keyname": {
+                "values": ["blacklist-*"],  # Match any key starting with 'blacklist-'
+                "isExcludes": False,
+                "isRecursive": False
+            }
+        },
+        "policyItems": [{
+            "accesses": [
+                {"type": "CREATE", "isAllowed": True},
+                {"type": "ROLLOVER", "isAllowed": True},
+                {"type": "DELETE", "isAllowed": True}
+            ],
+            "users": [user1]
+        }]
+    }
+
+    response = requests.post(BASE_URL_RANGER, auth=RANGER_AUTH, json=policy_data)
+    time.sleep(30)  # Wait for policy propagation

Review Comment:
   Will be resolved later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141355707


##########
PyTest-KMS-HDFS/pytest.ini:
##########
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time

Review Comment:
   The comments here are incorrect, review the comments in all files and update.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.

kumaab commented on code in PR #547:
URL: https://github.com/apache/ranger/pull/547#discussion_r2141364127


##########
PyTest-KMS-HDFS/readme.md:
##########
@@ -0,0 +1,94 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# This workflow will build a Java project with Maven, and cache/restore any dependencies to improve the workflow execution time
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-java-with-maven
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+
+
+#    KMS API & HDFS Encryption Pytest Suite
+
+
+This test suite validates REST API endpoints for KMS (Key Management Service) and tests HDFS encryption functionalities including key management and file operations within encryption zones.
+
+**test_kms  :** contains test cases for checking KMS API functionality  
+
+**test_hdfs :** contains test cases for checking KMS functionality through hdfs encryption lifecycle
+
+## 📂 Directory Structure
+
+```
+test_directory/
+├── test_kms/                # Tests on KMS API

Review Comment:
   Suggested directory structure:
   
       tests
       ├──hdfs
           ├── test_encryption.py
           ├── test_audit_spooling.py  
           ├── test_config.py         
           ├── conf-test.py           
           ├── readme.md
           ├── utils.py 
       ├── kms  
           ├── test_keys.py                  
           ├── test_blacklisting.py  
           ├── conf-test.py
           ├── utils.py 
           ├── readme.md



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption [ranger]

Posted by "ChinmayHegde24 (via GitHub)" <gi...@apache.org>.

ChinmayHegde24 commented on PR #547:
URL: https://github.com/apache/ranger/pull/547#issuecomment-4095602794

   It's a very old PR with stale code, closing this and opening new one with extra changes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org