You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/04/13 20:50:00 UTC
[jira] [Commented] (IMPALA-11232) Consider not adding some jars when starting HMS
[ https://issues.apache.org/jira/browse/IMPALA-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521943#comment-17521943 ]
ASF subversion and git services commented on IMPALA-11232:
----------------------------------------------------------
Commit c190966db932cf7b24881ce2213fd6f7d9192099 in impala's branch refs/heads/master from Fang-Yu Rao
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c190966db ]
IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS
This patch changes the line that added to HADOOP_CLASSPATH all the jar
files in the folder ${RANGER_HOME}/ews/webapp/WEB-INF/lib to a line that
only includes those jar files with names starting with "ranger-" since
almost all other jar files do not seem to be necessary to run the E2E
test of test_hive_with_ranger_setup.
This way we also avoid adding too many paths to HADOOP_CLASSPATH, which
in turn could result in Hadoop not being able to return its version to
the script that starts HMS due to the error of "Argument list too long".
Testing:
- Verified after this patch, test_hive_with_ranger_setup still
succeeds.
- Verified in a local development environment that the length of
Hadoop's environment variable 'CLASSPATH' logged in
hive-metastore.out decreases from 100,876 characters to 62,634
characters when executing run-hive-server.sh with the flag
'-with_ranger' if $HADOOP_SHELL_SCRIPT_DEBUG is "true" and
$IMPALA_HOME is "/home/fangyurao/Impala_for_FE".
Change-Id: Ifd66fd99a346835b9f81f95b5f046273fcce2590
Reviewed-on: http://gerrit.cloudera.org:8080/18398
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Consider not adding some jars when starting HMS
> -----------------------------------------------
>
> Key: IMPALA-11232
> URL: https://issues.apache.org/jira/browse/IMPALA-11232
> Project: IMPALA
> Issue Type: Test
> Components: Infrastructure
> Affects Versions: Impala 4.0.0
> Reporter: Fang-Yu Rao
> Assignee: Fang-Yu Rao
> Priority: Minor
>
> Currently when we execute {{run-hive-server.sh}} with the "{{-with_ranger}}" flag, we add all the jar files under the folder of {{$RANGER_HOME/ews/webapp/WEB-INF/lib}} to '{{HADOOP_CLASSPATH}}' before we call the shell script that starts the Hive service, i.e., {{bin/hive}}.
> Recall that {{bin/hive}} would attempt to execute '{{hadoop version}}' to determine the version of the underlying Hadoop distribution before starting the Hive Metastore and the HiveServer2. By default, this Hadoop command under the covers would add every path on '{{HADOOP_CLASSPATH}}' to its own environment variable of '{{CLASSPATH}}'. When there are too many paths added to '{{CLASSPATH}}', the Hadoop command would fail with the error message of "{{Argument list too long.}}" and hence the Hive service could not be started.
> We execute {{run-hive-server.sh}} with the "{{-with_ranger}}" flag in order to run the end-to-end test of {{test_hive_with_ranger_setup}}. We should consider not adding some (or all) of the jar files to '{{HADOOP_CLASSPATH}}' from the folder described if some (or all) of them are not required to run the end-to-end test so as to avoid the error mentioned above.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org