[impala] branch master updated: IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS

[wz...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wzhou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new c190966db IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS
c190966db is described below

commit c190966db932cf7b24881ce2213fd6f7d9192099
Author: Fang-Yu Rao <fa...@cloudera.com>
AuthorDate: Sun Apr 10 10:24:52 2022 -0700

    IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS
    
    This patch changes the line that added to HADOOP_CLASSPATH all the jar
    files in the folder ${RANGER_HOME}/ews/webapp/WEB-INF/lib to a line that
    only includes those jar files with names starting with "ranger-" since
    almost all other jar files do not seem to be necessary to run the E2E
    test of test_hive_with_ranger_setup.
    
    This way we also avoid adding too many paths to HADOOP_CLASSPATH, which
    in turn could result in Hadoop not being able to return its version to
    the script that starts HMS due to the error of "Argument list too long".
    
    Testing:
     - Verified after this patch, test_hive_with_ranger_setup still
       succeeds.
     - Verified in a local development environment that the length of
       Hadoop's environment variable 'CLASSPATH' logged in
       hive-metastore.out decreases from 100,876 characters to 62,634
       characters when executing run-hive-server.sh with the flag
       '-with_ranger' if $HADOOP_SHELL_SCRIPT_DEBUG is "true" and
       $IMPALA_HOME is "/home/fangyurao/Impala_for_FE".
    
    Change-Id: Ifd66fd99a346835b9f81f95b5f046273fcce2590
    Reviewed-on: http://gerrit.cloudera.org:8080/18398
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 testdata/bin/run-hive-server.sh | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/testdata/bin/run-hive-server.sh b/testdata/bin/run-hive-server.sh
index 9580bd648..909a302dd 100755
--- a/testdata/bin/run-hive-server.sh
+++ b/testdata/bin/run-hive-server.sh
@@ -90,15 +90,14 @@ suspend=n,address=30010"
 if [[ $ENABLE_RANGER_AUTH -eq 1 ]]; then
   export HIVE_CONF_DIR="$HADOOP_CONF_DIR/hive-site-ranger-auth/"
   for f in "$RANGER_HOME"/ews/webapp/WEB-INF/classes/ranger-plugins/hive/ranger-*.jar \
-      "$RANGER_HOME"/ews/webapp/WEB-INF/lib/*.jar \
+      "$RANGER_HOME"/ews/webapp/WEB-INF/lib/ranger-*.jar \
       "$RANGER_HOME"/ews/lib/ranger-*.jar; do
     FILE_NAME=$(basename $f)
-    # Exclude unneccessary jars.
-    if [[ ! $FILE_NAME == hive* && ! $FILE_NAME == hadoop* && ! $FILE_NAME == hbase* \
-        && ! $FILE_NAME == zookeeper* ]]; then
-      export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${f}
-    fi
+    export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${f}
   done
+  # The following jar is needed by RangerRESTUtils.java.
+  export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:\
+      ${RANGER_HOME}/ews/webapp/WEB-INF/lib/gethostname4j-*.jar"
 fi
 
 # For Hive 3, we use Tez for execution. We have to add it to the classpath.