You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Fredy Wijaya (Code Review)" <ge...@cloudera.org> on 2019/02/13 19:30:24 UTC

[Impala-ASF-CR] IMPALA-8099: Update the build scripts to support Apache Ranger

Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12469


Change subject: IMPALA-8099: Update the build scripts to support Apache Ranger
......................................................................

IMPALA-8099: Update the build scripts to support Apache Ranger

This patch updates the build scripts to suport Apache Ranger:
- Download Apache Ranger
- Setup Apache Ranger database
- Create Apache Ranger configuration files
- Start/stop Apache Ranger

Testing:
- Ran ./buildall.sh -format on a clean repository and was able to start
  Ranger without any problem.

Change-Id: I249cd64d74518946829e8588ed33d5ac454ffa7b
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M buildall.sh
M testdata/bin/kill-all.sh
A testdata/bin/kill-ranger-server.sh
M testdata/bin/run-all.sh
A testdata/bin/run-ranger-server.sh
A testdata/cluster/ranger/core-site.xml
A testdata/cluster/ranger/install.properties.template
A testdata/cluster/ranger/java_home.sh
A testdata/cluster/ranger/ranger-admin-default-site.xml.template
A testdata/cluster/ranger/ranger-admin-env-hadoopconfdir.sh
A testdata/cluster/ranger/ranger-admin-env-logdir.sh
A testdata/cluster/ranger/ranger-admin-env-piddir.sh
A testdata/cluster/ranger/ranger-admin-site.xml.template
A testdata/cluster/ranger/security-applicationContext.xml
17 files changed, 1,402 insertions(+), 30 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12469/2
-- 
To view, visit http://gerrit.cloudera.org:8080/12469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I249cd64d74518946829e8588ed33d5ac454ffa7b
Gerrit-Change-Number: 12469
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-8099: Update the build scripts to support Apache Ranger

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12469 )

Change subject: IMPALA-8099: Update the build scripts to support Apache Ranger
......................................................................


Patch Set 2:

(8 comments)

If you have a chance, I'd love for you to see that test-with-docker doesn't bail out with this change.

No major issues; just some questions below.

http://gerrit.cloudera.org:8080/#/c/12469/2/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:

http://gerrit.cloudera.org:8080/#/c/12469/2/bin/create-test-configuration.sh@204
PS2, Line 204: p -f "${RANGER_TEST_CONF_DIR}/java_home.sh" "${RANGER_SERVER_CONF_DIR}"
             : cp -f "${RANGER_TEST_CONF_DIR}/ranger-admin-env-hadoopconfdir.sh" \
             :     "${RANGER_SERVER_CONF_DIR}"
             : cp -f "${RANGER_TEST_CONF_DIR}/ranger-admin-env-logdir.sh" \
             :     "${RANGER_SERVER_CONF_DIR}"
             : cp -f "${RANGER_TEST_CONF_DIR}/ranger-admin-env-piddir.sh" \
             :     "${RANGER_SERVER_CONF_DIR}"
             : cp -f "${RANGER_TEST_CONF_DIR}/security-applicationContext.xml" \
             :     "${RANGER_SERVER_CONF_DIR}"
             : cp -f "${RANGER_TEST_CONF_DIR}/core-site.xml" "${RANGER_SERVER_CONF_DIR}"
Is this "cp ${RANGE_TEST_CONF_DIR}/* $RANGER_SERVER_CONF_DIR"?


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/bin/run-ranger-server.sh
File testdata/bin/run-ranger-server.sh:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/bin/run-ranger-server.sh@31
PS2, Line 31: JAVA_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30130" \
Is it intentional that this has debug ports open by default?


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/bin/run-ranger-server.sh@32
PS2, Line 32:     "${RANGER_HOME}"/ews/ranger-admin-services.sh restart
Does this have an XmX setting? If we don't, these things eat a ton of RAM on machines with a lot of RAM.


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/install.properties.template
File testdata/cluster/ranger/install.properties.template:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/install.properties.template@37
PS2, Line 37: #SQL_CONNECTOR_JAR=/opt/sqlanywhere17/java/sajdbc4.jar
This is super noisy with a lot of things commented out.


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/java_home.sh
File testdata/cluster/ranger/java_home.sh:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/java_home.sh@1
PS2, Line 1: export JAVA_HOME=${JAVA_HOME}
Why is this necessary? We have JAVA_HOME configured anyway...


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-default-site.xml.template
File testdata/cluster/ranger/ranger-admin-default-site.xml.template:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-default-site.xml.template@510
PS2, Line 510:     <value>tzL1AKl5uc4NKYaoQ4P3WLGIBFPXWPWdu1fRm9004jtQiV</value>
Is this a problem?


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-env-hadoopconfdir.sh
File testdata/cluster/ranger/ranger-admin-env-hadoopconfdir.sh:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-env-hadoopconfdir.sh@1
PS2, Line 1: export RANGER_HADOOP_CONF_DIR=/etc/hadoop/conf
Don't we need to substitute this?


http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-env-logdir.sh
File testdata/cluster/ranger/ranger-admin-env-logdir.sh:

http://gerrit.cloudera.org:8080/#/c/12469/2/testdata/cluster/ranger/ranger-admin-env-logdir.sh@1
PS2, Line 1: export RANGER_ADMIN_LOG_DIR=${RANGER_HOME}/ews/logs
We put sentry logs somewhere else, I think...



-- 
To view, visit http://gerrit.cloudera.org:8080/12469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I249cd64d74518946829e8588ed33d5ac454ffa7b
Gerrit-Change-Number: 12469
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Feb 2019 19:46:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8099: Update the build scripts to support Apache Ranger

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.

Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12469 )

Change subject: IMPALA-8099: Update the build scripts to support Apache Ranger
......................................................................


Patch Set 3:

I ran all tests with test-with-docker and they all passed.


-- 
To view, visit http://gerrit.cloudera.org:8080/12469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I249cd64d74518946829e8588ed33d5ac454ffa7b
Gerrit-Change-Number: 12469
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward (402)
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Feb 2019 02:10:45 +0000
Gerrit-HasComments: No