You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "fmorg-git (via GitHub)" <gi...@apache.org> on 2026/03/11 01:26:00 UTC
[PR] HDDS-14353. [STS] Part 1 - STS Ranger Smoke Tests [ozone]
fmorg-git opened a new pull request, #9901:
URL: https://github.com/apache/ozone/pull/9901
Please describe your PR in detail:
* This PR is the first step to adding integration smoke tests for STS tokens with Ranger. It builds on the initial framework by @adoroszlai. The scripts were updated to support either full Apache Ranger releases or Apache Ranger snapshot build in case a developer wants to work with a snapshot version of Ranger. Follow on PRs will actually include the smoke tests.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14353
## How was this patch tested?
updating local branch with requisite changes and running the smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14353. [STS] Part 1 - STS Ranger Smoke Tests [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git closed pull request #9901: HDDS-14353. [STS] Part 1 - STS Ranger Smoke Tests
URL: https://github.com/apache/ozone/pull/9901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14353. [STS] Part 1 - STS Ranger Smoke Tests [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9901:
URL: https://github.com/apache/ozone/pull/9901#discussion_r2916011637
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/.env:
##########
@@ -24,7 +24,7 @@ OZONE_VOLUME=./data
OZONE_OPTS=
RANGER_DB_IMAGE=postgres
RANGER_DB_IMAGE_VERSION=12
-RANGER_IMAGE=ghcr.io/adoroszlai/ranger-admin
-RANGER_IMAGE_VERSION=0ae34250d3af672776fca6a53047699adf3afce5-${ranger.version}-8
+RANGER_IMAGE=ghcr.io/fmorg-git/ranger-admin
+RANGER_IMAGE_VERSION=5c567041d56bf153a1c7fb320af42b4a9af673b0-${ranger.version}-8
Review Comment:
We have already switched to official Ranger images on `master`, since they are now available for both amd64 and arm64. (Previously it was arm64-only, which does not work in CI.)
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh:
##########
@@ -31,20 +31,79 @@ export OM_SERVICE_ID="omservice"
export SCM=scm1.org
export SECURITY_ENABLED=true
-if [[ "${SKIP_APACHE_VERIFY_DOWNLOAD}" != "true" ]]; then
+# Check if we are using a snapshot version
+if [[ "${RANGER_VERSION}" =~ [0-9]+\.[0-9]+\.[0-9]+-[0-9]{8}\.[0-9]{6}-[0-9]+ ]] || [[ "${RANGER_VERSION}" == *"SNAPSHOT"* ]]; then
+ IS_SNAPSHOT=true
+else
+ IS_SNAPSHOT=false
+fi
+
+if [[ "${SKIP_APACHE_VERIFY_DOWNLOAD}" != "true" ]] && [[ "${IS_SNAPSHOT}" == "false" ]]; then
curl -LO https://downloads.apache.org/ranger/KEYS
gpg --import KEYS
fi
-download_and_verify_apache_release "ranger/${RANGER_VERSION}/apache-ranger-${RANGER_VERSION}.tar.gz"
-tar -C "${DOWNLOAD_DIR}" -x -z -f "${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}.tar.gz"
-export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}"
+if [[ "${IS_SNAPSHOT}" == "true" ]]; then
+ # Snapshot download logic
+ RANGER_BASE_VERSION=$(echo "${RANGER_VERSION}" | sed -E 's/-[0-9]{8}\.[0-9]{6}-[0-9]+//')
+ if [[ "${RANGER_BASE_VERSION}" == "${RANGER_VERSION}" ]]; then
+ RANGER_BASE_VERSION="${RANGER_VERSION}"
+ else
+ RANGER_BASE_VERSION="${RANGER_BASE_VERSION}-SNAPSHOT"
+ fi
+ SNAPSHOT_REPO="https://repository.apache.org/content/groups/snapshots/org/apache/ranger/ranger-distro/${RANGER_BASE_VERSION}"
+
+ if [[ "${RANGER_VERSION}" == *"SNAPSHOT"* ]]; then
+ # If RANGER_VERSION is a snapshot (e.g. 2.8.0-SNAPSHOT), resolve it to the latest timestamped version
+ download_if_not_exists "${SNAPSHOT_REPO}/maven-metadata.xml" "${DOWNLOAD_DIR}/maven-metadata.xml"
+ TIMESTAMP=$(grep "<timestamp>" "${DOWNLOAD_DIR}/maven-metadata.xml" | head -1 | sed -e 's/.*<timestamp>\(.*\)<\/timestamp>.*/\1/')
+ BUILDNUM=$(grep "<buildNumber>" "${DOWNLOAD_DIR}/maven-metadata.xml" | head -1 | sed -e 's/.*<buildNumber>\(.*\)<\/buildNumber>.*/\1/')
+ if [[ -n "${TIMESTAMP}" ]] && [[ -n "${BUILDNUM}" ]]; then
+ RANGER_VERSION="${RANGER_BASE_VERSION%-SNAPSHOT}-${TIMESTAMP}-${BUILDNUM}"
+ echo "Resolved RANGER_VERSION to ${RANGER_VERSION}"
+ fi
+ fi
+
+ SRC_TAR="ranger-distro-${RANGER_VERSION}-src.tar.gz"
+ download_if_not_exists "${SNAPSHOT_REPO}/${SRC_TAR}" "${DOWNLOAD_DIR}/${SRC_TAR}"
+ tar -C "${DOWNLOAD_DIR}" -x -z -f "${DOWNLOAD_DIR}/${SRC_TAR}"
+
+ # Find the extracted directory name
+ EXTRACTED_DIR=$(tar -tf "${DOWNLOAD_DIR}/${SRC_TAR}" | grep -o '^[^/]*' | sort | uniq | head -1)
+ export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/${EXTRACTED_DIR}"
+else
+ # Release download logic
+ download_and_verify_apache_release "ranger/${RANGER_VERSION}/apache-ranger-${RANGER_VERSION}.tar.gz"
+ tar -C "${DOWNLOAD_DIR}" -x -z -f "${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}.tar.gz"
+ export RANGER_SOURCE_DIR="${DOWNLOAD_DIR}/apache-ranger-${RANGER_VERSION}"
+fi
+
chmod -R a+rX "${RANGER_SOURCE_DIR}"
-chmod a+x "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/config/*.sh
-download_and_verify_apache_release "ranger/${RANGER_VERSION}/plugins/ozone/ranger-${RANGER_VERSION}-ozone-plugin.tar.gz"
-tar -C "${DOWNLOAD_DIR}" -x -z -f "${DOWNLOAD_DIR}/ranger-${RANGER_VERSION}-ozone-plugin.tar.gz"
-export RANGER_OZONE_PLUGIN_DIR="${DOWNLOAD_DIR}/ranger-${RANGER_VERSION}-ozone-plugin"
+# Ranger docker support scripts moved between releases (eg: from config/*.sh to scripts/**).
+# Ensure we don't fail if a glob doesn't match, but still make init scripts executable when present.
+shopt -s nullglob
+chmod_targets=(
+ "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/config/*.sh
+ "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker/scripts/rdbms/*.sh
+)
+shopt -u nullglob
+if (( ${#chmod_targets[@]} > 0 )); then
+ chmod a+x "${chmod_targets[@]}"
+fi
Review Comment:
This can be simplified:
```bash
find "${RANGER_SOURCE_DIR}"/dev-support/ranger-docker -name '*.sh' | xargs --no-run-if-empty chmod a+x
```
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh:
##########
@@ -53,6 +112,8 @@ perl -wpl -i \
-e 's@^POLICY_MGR_URL=.*@POLICY_MGR_URL=http://ranger:6080@;' \
-e 's@^REPOSITORY_NAME=.*@REPOSITORY_NAME=dev_ozone@;' \
-e 's@^CUSTOM_USER=ozone@CUSTOM_USER=hadoop@;' \
+ -e 's@^XAAUDIT.LOG4J.ENABLE=.*@XAAUDIT.LOG4J.ENABLE=false@;' \
+ -e 's@^XAAUDIT.LOG4J.DESTINATION.LOG4J=.*@XAAUDIT.LOG4J.DESTINATION.LOG4J=false@;' \
Review Comment:
This was already added as part of bump to Ranger 2.7.0 (42c7ff52690a8ffa99fd4de60f7d6ed07466af1f).
##########
pom.xml:
##########
@@ -197,7 +197,7 @@
<protobuf-maven-plugin.version>0.6.1</protobuf-maven-plugin.version>
<protobuf2.version>2.5.0</protobuf2.version>
<protobuf3.version>3.25.8</protobuf3.version>
- <ranger.version>2.6.0</ranger.version>
+ <ranger.version>2.8.0</ranger.version>
Review Comment:
Ranger 2.8.0 switched from `hadoop-common` to `hadoop-client-api`, we need to exclude the new dependencies to avoid [build failure](https://github.com/fmorg-git/ozone/actions/runs/22932059910/job/66555565363#step:16:17):
```diff
diff --git hadoop-ozone/multitenancy-ranger/pom.xml hadoop-ozone/multitenancy-ranger/pom.xml
index 526171a750..9d22925d2e 100644
--- hadoop-ozone/multitenancy-ranger/pom.xml
+++ hadoop-ozone/multitenancy-ranger/pom.xml
@@ -73,6 +73,14 @@
<groupId>net.minidev</groupId>
<artifactId>json-smart</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-storage-api</artifactId>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14353. [STS] Part 1 - STS Ranger Smoke Tests [ozone]
Posted by "fmorg-git (via GitHub)" <gi...@apache.org>.
fmorg-git commented on PR #9901:
URL: https://github.com/apache/ozone/pull/9901#issuecomment-4058458841
Closing this PR as the Ranger 2.8.0 upgrade work was done in master (https://issues.apache.org/jira/browse/HDDS-14813) and master was merged into this feature branch. Here is the PR for master: https://github.com/apache/ozone/pull/9909. Remaining artifacts were smoke-test related and included in https://github.com/apache/ozone/pull/9902/changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org