You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "kumaab (via GitHub)" <gi...@apache.org> on 2024/08/25 23:19:23 UTC

[PR] RANGER-3801: Add support for Ozone in docker [ranger]

kumaab opened a new pull request, #377:
URL: https://github.com/apache/ranger/pull/377

   ## What changes were proposed in this pull request?
   - Adds support for ozone services in ranger docker environment with ranger-plugin enabled for OzoneManager (OM) container.
   - A minimal setup for Ozone requires 1 OM, 1 SCM and 1 DataNode (see `docker-compose.ranger-ozone.yml`)
   - Updates the README for docker based setup to include Ozone.
   - Re-factor of download-archives.sh to allow specific downloads.
   - Creation of ozone service during ranger-admin startup.
   
   
   
   ## How was this patch tested?
   - Verified `setup_ozone`
   - Verified containers come up with `docker-compose -f docker-compose.ranger.yml -f docker-compose.ranger-postgres.yml -f docker-compose.ranger-ozone.yml up -d`
   - Verified Ozone plugin installation in `om` container
   - Verified restart of `om` container, plugin installation is skipped!
   - Audits are flowing to Solr
   - Verified allow policy on a volume works for trino user.
   - Verified allow policy works for om user.
   - Verified deny works for root user.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.
kumaab commented on code in PR #377:
URL: https://github.com/apache/ranger/pull/377#discussion_r1753007519


##########
dev-support/ranger-docker/Dockerfile.ranger-ozone:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG OZONE_RUNNER_IMAGE
+ARG OZONE_RUNNER_VERSION
+FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+
+ARG OZONE_HOME
+ARG OZONE_PLUGIN_VERSION
+
+USER root
+
+RUN useradd -g hadoop -ms /bin/bash ozone

Review Comment:
   Dockerfile.ranger-base and Dockerfile.ranger-ozone refer to different independent images so adding it here as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.
kumaab merged PR #377:
URL: https://github.com/apache/ranger/pull/377


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.
kumaab commented on code in PR #377:
URL: https://github.com/apache/ranger/pull/377#discussion_r1753008230


##########
dev-support/ranger-docker/Dockerfile.ranger-ozone:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG OZONE_RUNNER_IMAGE
+ARG OZONE_RUNNER_VERSION
+FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+
+ARG OZONE_HOME
+ARG OZONE_PLUGIN_VERSION
+
+USER root
+
+RUN useradd -g hadoop -ms /bin/bash ozone
+RUN mkdir -p -m 755 /var/log/ozone
+RUN chown om:om -R /var/log/ozone

Review Comment:
   Removed this, currently the process runs as root, will follow-up on this in another jira to run OzoneManager as hadoop.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "mneethiraj (via GitHub)" <gi...@apache.org>.
mneethiraj commented on code in PR #377:
URL: https://github.com/apache/ranger/pull/377#discussion_r1747539323


##########
dev-support/ranger-docker/README.md:
##########
@@ -30,9 +30,13 @@ Docker files in this folder create docker images and run them to build Apache Ra
 
 3. Set ```dev-support/ranger-docker``` as your working directory.
 
-4. Execute following command to download necessary archives to setup Ranger/HDFS/Hive/HBase/Kafka/Knox services:
+4. Execute following command to download necessary archives to setup Ranger/HDFS/Hive/HBase/Kafka/Knox/Ozone services:
    ~~~
-   chmod +x download-archives.sh && ./download-archives.sh
+   chmod +x download-archives.sh
+   source download-archives.sh
+   
+   # use a subset of the below to download specific services
+   download_hadoop; download_hbase; download_hive; download_kafka; download_knox; download_ozone

Review Comment:
   above `source` command exports methods defined in download-archives.sh - `download_hadoop()`, `download_hbase()`, `download_..()`. These methods don't need to be available in the shell. Instead of this approach, consider updating `download-archives.sh` to take the list of components to download as arguments, as shown below:
   
   `./download-archives.sh hadoop hbase kafka`



##########
dev-support/ranger-docker/Dockerfile.ranger-ozone:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG OZONE_RUNNER_IMAGE
+ARG OZONE_RUNNER_VERSION
+FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+
+ARG OZONE_HOME
+ARG OZONE_PLUGIN_VERSION
+
+USER root
+
+RUN useradd -g hadoop -ms /bin/bash ozone
+RUN mkdir -p -m 755 /var/log/ozone
+RUN chown om:om -R /var/log/ozone

Review Comment:
   Shouldn't the username be `ozone`, instead of `om` - refer to #26 above?



##########
dev-support/ranger-docker/Dockerfile.ranger-ozone:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG OZONE_RUNNER_IMAGE
+ARG OZONE_RUNNER_VERSION
+FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+
+ARG OZONE_HOME
+ARG OZONE_PLUGIN_VERSION
+
+USER root
+
+RUN useradd -g hadoop -ms /bin/bash ozone

Review Comment:
   User `ozone` is already created in Dockerfile.ranger-base. Is it necessary to create here as well? Does this fail if the user already exists?



##########
dev-support/ranger-docker/README.md:
##########
@@ -42,44 +46,66 @@ Docker files in this folder create docker images and run them to build Apache Ra
    export RANGER_DB_TYPE=postgres
    ~~~
 
-6. Build Apache Ranger in containers using docker-compose
-
-   1. Execute following command to build Apache Ranger:
-      ~~~
-      docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger-build.yml up
-      ~~~
-
-      Time taken to complete the build might vary (upto an hour), depending on status of ```${HOME}/.m2``` directory cache.
-
-   2. Alternatively, the following commands can be executed from the parent directory
-      1. To generate tarballs:```mvn clean package -DskipTests```
-
-      2. Copy the tarballs and version file to ```dev-support/ranger-docker/dist```
-         ~~~
-         cp target/ranger-* dev-support/ranger-docker/dist/
-         cp target/version dev-support/ranger-docker/dist/
-         ~~~
-
-      3. Build the ranger-base image:
-         ~~~
-         docker-compose -f docker-compose.ranger-base.yml build --no-cache
-         ~~~
-7. To enable file based sync source for usersync execute: ```export ENABLE_FILE_SYNC_SOURCE=true```
-
-8. Execute following command to start Ranger, Ranger Usersync, Ranger Tagsync, Ranger enabled HDFS/YARN/HBase/Hive/Kafka/Knox and dependent services (Solr, DB) in containers:
-   ~~~
-   docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger.yml -f docker-compose.ranger-${RANGER_DB_TYPE}.yml -f docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml -f docker-compose.ranger-kms.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-hbase.yml -f docker-compose.ranger-kafka.yml -f docker-compose.ranger-hive.yml -f docker-compose.ranger-knox.yml up -d
-   ~~~
-
-	- valid values for RANGER_DB_TYPE: mysql or postgres
-9. To run ranger enabled Trino in containers (Requires docker build with JDK 11):
-   ~~~
-   docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger.yml -f docker-compose.ranger-${RANGER_DB_TYPE}.yml -f docker-compose.ranger-trino.yml up -d
-   ~~~
-
-10. To rebuild specific images and start containers with the new image, use following command:
-   ~~~
-   docker-compose -f docker-compose.ranger-base.yml -f docker-compose.ranger.yml -f docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml -f docker-compose.ranger-kms.yml -f docker-compose.ranger-hadoop.yml -f docker-compose.ranger-hbase.yml -f docker-compose.ranger-kafka.yml -f docker-compose.ranger-hive.yml -f docker-compose.ranger-trino.yml -f docker-compose.ranger-knox.yml up -d --no-deps --force-recreate --build <service-1> <service-2>
-   ~~~
-
-9. Ranger Admin can be accessed at http://localhost:6080 (admin/rangerR0cks!)
+### Apache Ranger Build

Review Comment:
   I suggest to apply the reformatting for earlier steps as well (1 to 5).



##########
dev-support/ranger-docker/download-archives.sh:
##########
@@ -43,14 +41,49 @@ downloadIfNotPresent() {
   fi
 }
 
-downloadIfNotPresent hadoop-${HADOOP_VERSION}.tar.gz        https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}
-downloadIfNotPresent hbase-${HBASE_VERSION}-bin.tar.gz      https://archive.apache.org/dist/hbase/${HBASE_VERSION}
-downloadIfNotPresent kafka_2.12-${KAFKA_VERSION}.tgz        https://archive.apache.org/dist/kafka/${KAFKA_VERSION}
-downloadIfNotPresent apache-hive-${HIVE_VERSION}-bin.tar.gz https://archive.apache.org/dist/hive/hive-${HIVE_VERSION}
-downloadIfNotPresent hadoop-${HIVE_HADOOP_VERSION}.tar.gz   https://archive.apache.org/dist/hadoop/common/hadoop-${HIVE_HADOOP_VERSION}
-downloadIfNotPresent postgresql-42.2.16.jre7.jar            https://search.maven.org/remotecontent?filepath=org/postgresql/postgresql/42.2.16.jre7
-downloadIfNotPresent mysql-connector-java-8.0.28.jar        https://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/8.0.28
+downloadIfNotPresent postgresql-42.2.16.jre7.jar            "https://search.maven.org/remotecontent?filepath=org/postgresql/postgresql/42.2.16.jre7"
+downloadIfNotPresent mysql-connector-java-8.0.28.jar        "https://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/8.0.28"
 downloadIfNotPresent log4jdbc-1.2.jar                       https://repo1.maven.org/maven2/com/googlecode/log4jdbc/log4jdbc/1.2
 
-downloadIfNotPresent knox-${KNOX_VERSION}.tar.gz            https://archive.apache.org/dist/knox/${KNOX_VERSION}
+download_hadoop() {
+  downloadIfNotPresent hadoop-${HADOOP_VERSION}.tar.gz        https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}
+}
+
+download_hbase() {
+  downloadIfNotPresent hbase-${HBASE_VERSION}-bin.tar.gz      https://archive.apache.org/dist/hbase/${HBASE_VERSION}
+}
+
+download_hive() {
+  downloadIfNotPresent apache-hive-${HIVE_VERSION}-bin.tar.gz https://archive.apache.org/dist/hive/hive-${HIVE_VERSION}
+  downloadIfNotPresent hadoop-${HIVE_HADOOP_VERSION}.tar.gz   https://archive.apache.org/dist/hadoop/common/hadoop-${HIVE_HADOOP_VERSION}
+}
+
+download_kafka() {
+  downloadIfNotPresent kafka_2.12-${KAFKA_VERSION}.tgz        https://archive.apache.org/dist/kafka/${KAFKA_VERSION}
+}
+
+download_knox() {
+  downloadIfNotPresent knox-${KNOX_VERSION}.tar.gz            https://archive.apache.org/dist/knox/${KNOX_VERSION}
+}
+
+download_ozone() {
+    downloadIfNotPresent ozone-${OZONE_VERSION}.tar.gz        https://archive.apache.org/dist/ozone/${OZONE_VERSION}
+
+    if [ ! -d downloads/ozone-${OZONE_VERSION} ]
+    then
+      tar xvfz downloads/ozone-${OZONE_VERSION}.tar.gz --directory=downloads/
+    fi
+}
+
+setup_ozone() {

Review Comment:
   why is `setup_ozone()` in `download-archives.sh`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.
kumaab commented on code in PR #377:
URL: https://github.com/apache/ranger/pull/377#discussion_r1753007028


##########
dev-support/ranger-docker/download-archives.sh:
##########
@@ -43,14 +41,49 @@ downloadIfNotPresent() {
   fi
 }
 
-downloadIfNotPresent hadoop-${HADOOP_VERSION}.tar.gz        https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}
-downloadIfNotPresent hbase-${HBASE_VERSION}-bin.tar.gz      https://archive.apache.org/dist/hbase/${HBASE_VERSION}
-downloadIfNotPresent kafka_2.12-${KAFKA_VERSION}.tgz        https://archive.apache.org/dist/kafka/${KAFKA_VERSION}
-downloadIfNotPresent apache-hive-${HIVE_VERSION}-bin.tar.gz https://archive.apache.org/dist/hive/hive-${HIVE_VERSION}
-downloadIfNotPresent hadoop-${HIVE_HADOOP_VERSION}.tar.gz   https://archive.apache.org/dist/hadoop/common/hadoop-${HIVE_HADOOP_VERSION}
-downloadIfNotPresent postgresql-42.2.16.jre7.jar            https://search.maven.org/remotecontent?filepath=org/postgresql/postgresql/42.2.16.jre7
-downloadIfNotPresent mysql-connector-java-8.0.28.jar        https://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/8.0.28
+downloadIfNotPresent postgresql-42.2.16.jre7.jar            "https://search.maven.org/remotecontent?filepath=org/postgresql/postgresql/42.2.16.jre7"
+downloadIfNotPresent mysql-connector-java-8.0.28.jar        "https://search.maven.org/remotecontent?filepath=mysql/mysql-connector-java/8.0.28"
 downloadIfNotPresent log4jdbc-1.2.jar                       https://repo1.maven.org/maven2/com/googlecode/log4jdbc/log4jdbc/1.2
 
-downloadIfNotPresent knox-${KNOX_VERSION}.tar.gz            https://archive.apache.org/dist/knox/${KNOX_VERSION}
+download_hadoop() {
+  downloadIfNotPresent hadoop-${HADOOP_VERSION}.tar.gz        https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}
+}
+
+download_hbase() {
+  downloadIfNotPresent hbase-${HBASE_VERSION}-bin.tar.gz      https://archive.apache.org/dist/hbase/${HBASE_VERSION}
+}
+
+download_hive() {
+  downloadIfNotPresent apache-hive-${HIVE_VERSION}-bin.tar.gz https://archive.apache.org/dist/hive/hive-${HIVE_VERSION}
+  downloadIfNotPresent hadoop-${HIVE_HADOOP_VERSION}.tar.gz   https://archive.apache.org/dist/hadoop/common/hadoop-${HIVE_HADOOP_VERSION}
+}
+
+download_kafka() {
+  downloadIfNotPresent kafka_2.12-${KAFKA_VERSION}.tgz        https://archive.apache.org/dist/kafka/${KAFKA_VERSION}
+}
+
+download_knox() {
+  downloadIfNotPresent knox-${KNOX_VERSION}.tar.gz            https://archive.apache.org/dist/knox/${KNOX_VERSION}
+}
+
+download_ozone() {
+    downloadIfNotPresent ozone-${OZONE_VERSION}.tar.gz        https://archive.apache.org/dist/ozone/${OZONE_VERSION}
+
+    if [ ! -d downloads/ozone-${OZONE_VERSION} ]
+    then
+      tar xvfz downloads/ozone-${OZONE_VERSION}.tar.gz --directory=downloads/
+    fi
+}
+
+setup_ozone() {

Review Comment:
   Moved it to a different script, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] RANGER-3801: Add support for Ozone in docker [ranger]

Posted by "kumaab (via GitHub)" <gi...@apache.org>.
kumaab commented on code in PR #377:
URL: https://github.com/apache/ranger/pull/377#discussion_r1756066298


##########
dev-support/ranger-docker/Dockerfile.ranger-ozone:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG OZONE_RUNNER_IMAGE
+ARG OZONE_RUNNER_VERSION
+FROM ${OZONE_RUNNER_IMAGE}:${OZONE_RUNNER_VERSION}
+
+ARG OZONE_HOME
+ARG OZONE_PLUGIN_VERSION
+
+USER root
+
+RUN useradd -g hadoop -ms /bin/bash ozone
+RUN mkdir -p -m 755 /var/log/ozone
+RUN chown om:om -R /var/log/ozone

Review Comment:
   Created a jira here: https://issues.apache.org/jira/browse/RANGER-4929, thanks!
   CC: @mneethiraj 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org