You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2014/10/11 04:41:42 UTC
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/
-----------------------------------------------------------
(Updated Oct. 11, 2014, 2:41 a.m.)
Review request for Ambari, Andrew Onischuk, Dmytro Sen, and Sumit Mohanty.
Bugs: AMBARI-7739
https://issues.apache.org/jira/browse/AMBARI-7739
Repository: ambari
Description
-------
Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
... 8 more
***********************************************************
# This needs two +1s in order to be committed to branch-1.7.0
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
Diff: https://reviews.apache.org/r/26591/diff/
Testing
-------
Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
Then changed the following properties to coincide with the changed files.
HDFS => hadoop-env =
{
...
if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
# When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
fi
# Setting path to hdfs command line
export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
# Mostly required for hadoop 2.0
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
...
}
Hive => hive-env =
{
...
if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
fi
...
}
Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
Then ran
su - hive
hive
And was able to get the hive shell to start.
Also ran unit tests,
----------------------------------------------------------------------
Total run:641
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:21.678s
[INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
[INFO] Final Memory: 63M/1540M
[INFO] ------------------------------------------------------------------------
Thanks,
Alejandro Fernandez
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56278
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
<https://reviews.apache.org/r/26591/#comment96577>
Will create a separate Jira for this.
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml
<https://reviews.apache.org/r/26591/#comment96578>
Notice the paths
- Alejandro Fernandez
On Oct. 11, 2014, 2:41 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2014, 2:41 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, and Sumit Mohanty.
>
>
> Bugs: AMBARI-7739
> https://issues.apache.org/jira/browse/AMBARI-7739
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
>
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
>
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> ... 8 more
>
>
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
>
> Diff: https://reviews.apache.org/r/26591/diff/
>
>
> Testing
> -------
>
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
>
> HDFS => hadoop-env =
> {
> ...
>
> if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
> # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
>
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
>
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
>
> ...
> }
>
>
> Hive => hive-env =
> {
> ...
>
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
> export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
> export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
>
> ...
> }
>
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
>
>
> After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
>
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
>
> Then ran
> su - hive
> hive
>
> And was able to get the hive shell to start.
>
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56295
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
<https://reviews.apache.org/r/26591/#comment96608>
Yes it does.
And the test fails if the symlink exists but its target doesn't.
- Alejandro Fernandez
On Oct. 11, 2014, 3:23 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2014, 3:23 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
>
>
> Bugs: AMBARI-7739
> https://issues.apache.org/jira/browse/AMBARI-7739
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
>
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
>
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> ... 8 more
>
>
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
>
> Diff: https://reviews.apache.org/r/26591/diff/
>
>
> Testing
> -------
>
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
>
> HDFS => hadoop-env =
> {
> ...
>
> if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
> # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
>
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
>
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
>
> ...
> }
>
>
> Hive => hive-env =
> {
> ...
>
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
> export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
> export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
>
> ...
> }
>
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
>
>
> After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
>
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
>
> Then ran
> su - hive
> hive
>
> And was able to get the hive shell to start.
> To test this on a brand new cluster install, would like for this to be committed first.
>
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56391
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Sen
On Окт. 11, 2014, 3:23 д.п., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
>
> (Updated Окт. 11, 2014, 3:23 д.п.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
>
>
> Bugs: AMBARI-7739
> https://issues.apache.org/jira/browse/AMBARI-7739
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
>
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
>
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> ... 8 more
>
>
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
>
> Diff: https://reviews.apache.org/r/26591/diff/
>
>
> Testing
> -------
>
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
>
> HDFS => hadoop-env =
> {
> ...
>
> if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
> # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
>
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
>
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
>
> ...
> }
>
>
> Hive => hive-env =
> {
> ...
>
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
> export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
> export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
>
> ...
> }
>
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
>
>
> After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
>
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
>
> Then ran
> su - hive
> hive
>
> And was able to get the hive shell to start.
> To test this on a brand new cluster install, would like for this to be committed first.
>
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56282
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
<https://reviews.apache.org/r/26591/#comment96580>
Does "-d" check work for sym links?
- Sumit Mohanty
On Oct. 11, 2014, 3:23 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2014, 3:23 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
>
>
> Bugs: AMBARI-7739
> https://issues.apache.org/jira/browse/AMBARI-7739
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
>
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
>
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> ... 8 more
>
>
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
>
> Diff: https://reviews.apache.org/r/26591/diff/
>
>
> Testing
> -------
>
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
>
> HDFS => hadoop-env =
> {
> ...
>
> if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
> # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
>
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
>
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
>
> ...
> }
>
>
> Hive => hive-env =
> {
> ...
>
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
> export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
> export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
>
> ...
> }
>
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
>
>
> After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
>
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
>
> Then ran
> su - hive
> hive
>
> And was able to get the hive shell to start.
> To test this on a brand new cluster install, would like for this to be committed first.
>
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/#review56334
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Oct. 11, 2014, 3:23 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26591/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2014, 3:23 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
>
>
> Bugs: AMBARI-7739
> https://issues.apache.org/jira/browse/AMBARI-7739
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
>
> After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
>
> Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
> at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
> ... 8 more
>
>
> ***********************************************************
> # This needs two +1s in order to be committed to branch-1.7.0
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
>
> Diff: https://reviews.apache.org/r/26591/diff/
>
>
> Testing
> -------
>
> Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
> Then changed the following properties to coincide with the changed files.
>
> HDFS => hadoop-env =
> {
> ...
>
> if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
> # When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
> export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
> fi
>
> # Setting path to hdfs command line
> export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
>
> # Mostly required for hadoop 2.0
> export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
>
> ...
> }
>
>
> Hive => hive-env =
> {
> ...
>
> if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
> export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
> elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
> export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
> fi
>
> ...
> }
>
> Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
>
>
> After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
>
> export TEZ_CONF_DIR=/etc/tez/conf
> export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
> export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
>
> Then ran
> su - hive
> hive
>
> And was able to get the hive shell to start.
> To test this on a brand new cluster install, would like for this to be committed first.
>
> Also ran unit tests,
> ----------------------------------------------------------------------
> Total run:641
> Total errors:0
> Total failures:0
> OK
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:21.678s
> [INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
> [INFO] Final Memory: 63M/1540M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/
-----------------------------------------------------------
(Updated Oct. 11, 2014, 3:23 a.m.)
Review request for Ambari, Andrew Onischuk, Dmytro Sen, Hitesh Shah, and Sumit Mohanty.
Bugs: AMBARI-7739
https://issues.apache.org/jira/browse/AMBARI-7739
Repository: ambari
Description
-------
Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
... 8 more
***********************************************************
# This needs two +1s in order to be committed to branch-1.7.0
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
Diff: https://reviews.apache.org/r/26591/diff/
Testing
-------
Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
Then changed the following properties to coincide with the changed files.
HDFS => hadoop-env =
{
...
if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
# When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
fi
# Setting path to hdfs command line
export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
# Mostly required for hadoop 2.0
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
...
}
Hive => hive-env =
{
...
if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
fi
...
}
Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
Then ran
su - hive
hive
And was able to get the hive shell to start.
To test this on a brand new cluster install, would like for this to be committed first.
Also ran unit tests,
----------------------------------------------------------------------
Total run:641
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:21.678s
[INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
[INFO] Final Memory: 63M/1540M
[INFO] ------------------------------------------------------------------------
Thanks,
Alejandro Fernandez
Re: Review Request 26591: hive command line fails to start after
setting hive.execution.engine to tez in HDP 2.2.1
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26591/
-----------------------------------------------------------
(Updated Oct. 11, 2014, 2:43 a.m.)
Review request for Ambari, Andrew Onischuk, Dmytro Sen, and Sumit Mohanty.
Bugs: AMBARI-7739
https://issues.apache.org/jira/browse/AMBARI-7739
Repository: ambari
Description
-------
Create an Ambari 1.7.0 cluster with HDP 2.2.1, and install HDFS, Tez, MR, Hive, Zookeeper.
After switching the hive.execution.engine from "mr" to "tez", attempting to run the hive shell generates an error.
Logging initialized using configuration in file:/etc/hive/conf.server/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:457)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:598)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:206)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:119)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:454)
... 8 more
***********************************************************
# This needs two +1s in order to be committed to branch-1.7.0
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py 62e3cd6
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 93ad95d
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-env.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/hive-site.xml c1e8fa6
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml dfedb6b
Diff: https://reviews.apache.org/r/26591/diff/
Testing (updated)
-------
Deployed an HDP 2.2.1 cluster and switched the execution engine to tez.
Then changed the following properties to coincide with the changed files.
HDFS => hadoop-env =
{
...
if [[ -d "/usr/hdp/current/tez-client" && -d "/etc/tez/conf/" ]]; then
# When using versioned RPMs, the tez-client will be a symlink to the current folder of tez in HDP.
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/
fi
# Setting path to hdfs command line
export HADOOP_LIBEXEC_DIR={{hadoop_libexec_dir}}
# Mostly required for hadoop 2.0
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
...
}
Hive => hive-env =
{
...
if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog
fi
...
}
Tez => tez.lib.uris = hdfs:///apps/tez/tez.tar.gz
After restarting the services, ran the following commands to simulate the behavior of the scripts during the installation of Hadoop.
export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_JARS=/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*
export HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS:$HADOOP_CLASSPATH
Then ran
su - hive
hive
And was able to get the hive shell to start.
To test this on a brand new cluster install, would like for this to be committed first.
Also ran unit tests,
----------------------------------------------------------------------
Total run:641
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:21.678s
[INFO] Finished at: Fri Oct 10 19:39:55 PDT 2014
[INFO] Final Memory: 63M/1540M
[INFO] ------------------------------------------------------------------------
Thanks,
Alejandro Fernandez