You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "BsoBird (via GitHub)" <gi...@apache.org> on 2024/12/30 07:51:22 UTC
[PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
BsoBird opened a new pull request, #24:
URL: https://github.com/apache/hive-site/pull/24
https://issues.apache.org/jira/browse/HIVE-28683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899401414
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-SNAPSHOT-bin
Review Comment:
@deniskuzZ This is just a directory in our production environment;Because we directly compile the code to obtain the tar package, its version is a SNAPSHOT version. I'm just illustrating how to install. should i change this to non-snapshot version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899400719
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
Review Comment:
@deniskuzZ This is just a directory in our production environment; I'm just illustrating how to install.
`hmsclient01 ` => our production environment hostname
`opt folder` => our production environment dir
should i change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900150214
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
Review Comment:
@zhangbutao No, Sir. The value of tez.lib.uris can be a regular Unix file system path. TEZ will use this PATH to locate the libraries on HDFS, so there is no need to fill in a complete HDFS URL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899401414
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-SNAPSHOT-bin
Review Comment:
@deniskuzZ This is just a directory in our production environment; I'm just illustrating how to install. should i change this to non-snapshot version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900150214
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
Review Comment:
@zhangbutao No, Sir. The value of tez.lib.uris is a regular Unix file system path. TEZ will use this PATH to locate the libraries on HDFS, so there is no need to fill in a complete HDFS URL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900130182
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,77 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+#HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client}
Review Comment:
@zhangbutao sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #24:
URL: https://github.com/apache/hive-site/pull/24#issuecomment-2577883416
> @BsoBird I double check the link : https://hive.apache.org/docs/latest/manual-installation_283118363/ Foud that https://hive.apache.org/docs/latest/manual-installation_283118363/#installing-with-old-version-hadoop(greater-than-or-equal-3.1.0) can not be linked. Could you please fix this?
sure.i'll fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900139319
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
+ </property>
+ <property>
+ <name>tez.lib.uris.classpath</name> <!--only use tez self lib-->
+ <value>$PWD/tezlib/*,$PWD/tezlib/lib/*</value>
+ </property>
+
+ <property>
+ <name>tez.am.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
Review Comment:
Can remove the property?
If this configuration is required, please avoid using the hdp version.
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
Review Comment:
Please use a correct hdfs path format, such as `${fs.defaultFS}/apps/apache-tez-0.10.4-bin.tar.gz` (Refer to https://tez.apache.org/install.html)
BTW, should the apache-tez-0.10.4-bin.tar.gz contain all high hadoop version (3.3.6) related dependencies?
You can add a description for the property.
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
Review Comment:
You can add more about how to` Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies`. Such as what mvn command can be used to compile?
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
+ </property>
+ <property>
+ <name>tez.lib.uris.classpath</name> <!--only use tez self lib-->
+ <value>$PWD/tezlib/*,$PWD/tezlib/lib/*</value>
+ </property>
+
+ <property>
+ <name>tez.am.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
+
+ <property>
+ <name>tez.task.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
Review Comment:
Can remove the property?
If this configuration is required, please avoid using the hdp version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #24:
URL: https://github.com/apache/hive-site/pull/24#issuecomment-2577775220
@BsoBird I double check the link : https://hive.apache.org/docs/latest/manual-installation_283118363/
Foud that https://hive.apache.org/docs/latest/manual-installation_283118363/#installing-with-old-version-hadoop(greater-than-or-equal-3.1.0) can not be linked. Could you please fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899400719
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
Review Comment:
@deniskuzZ This is just a directory in our production environment; I'm just illustrating how to install.
`hmsclient01 ` => our production environment hostname
`opt folder` => our production environment dir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #24:
URL: https://github.com/apache/hive-site/pull/24#issuecomment-2566472781
If there is no difference between installing Hadoop3.1 and installing Hadoop3.3.6, I don't think we need to write the detailed configuration.
We just need to add a note that **Hive 4.0.1can be deployed safely against Hadoop version >3.1**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899400719
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
Review Comment:
This is just a directory in our production environment; I'm just illustrating how to install.
`hmsclient01 ` => hostname
`opt folder` => our production environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900148997
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop-3.3.6}
+
+export HIVE_HOME=${HIVE_HOME:-/opt/hive-4.0.0}
+```
+
+3.In tez-site.xml. Set the following two confs to use only the libs that come with tez. For nativeLib,
+you can reuse the cluster's existing libs.
+```xml
+ <property>
+ <name>tez.lib.uris</name><!--hdfs path-->
+ <value>/{hdfs-dir}/apache-tez-0.10.4-bin.tar.gz</value>
+ </property>
+ <property>
+ <name>tez.lib.uris.classpath</name> <!--only use tez self lib-->
+ <value>$PWD/tezlib/*,$PWD/tezlib/lib/*</value>
+ </property>
+
+ <property>
+ <name>tez.am.launch.env</name><!--Example, replace with actual value-->
+ <value>LD_LIBRARY_PATH=/usr/hdp/3.1.0.0-78/hadoop/lib/native</value>
+ </property>
Review Comment:
Okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900372758
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
Review Comment:
> You can add more about how to` Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies`. Such as what mvn command can be used to compile?
@done
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,76 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies(not tez minimal tarball),
Review Comment:
> You can add more about how to` Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies`. Such as what mvn command can be used to compile?
@zhangbutao done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #24:
URL: https://github.com/apache/hive-site/pull/24#issuecomment-2566482775
> If there is no difference in configuration/steps between installing Hive 4.0.1 on Hadoop 3.1 and on Hadoop 3.3.6, I don't think we need to write the detailed configuration. We just need to add a note that **Hive 4.0.1can be deployed safely against Hadoop version >3.1**.
@zhangbutao
No, Sir. Unlike the normal deployment, in this step we add all dependencies of Hadoop 3.3.6+ in HIVE & TEZ because if we deploy HIVE4 in a lower version Hadoop environment, we won't be able to use the jar packages and other resources from the lower version Hadoop environment.
Additionally, while experienced developers might be able to explore deployment solutions, many users do not have the relevant experience. Moreover, both the HIVE deployment documentation and the official HIVE website imply that Hadoop 3.3.6 is required for deploying HIVE4. Therefore, to increase the adoption rate of HIVE4, I believe we still need to provide a separate explanation for this matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900373575
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -16,6 +16,7 @@ date: 2024-12-12
+ [Extra hadoop configurations to make everything working]({{< ref "#extra-hadoop-configurations-to-make-everything-working" >}})
+ [Installing Hive from a Tarball]({{< ref "#installing-hive-from-a-tarball" >}})
+ [Installing from Source Code]({{< ref "#installing-from-source-code" >}})
+ + [Installing with old version hadoop(greater than or equal 3.1.0)]({{< ref "#installing-with-old-version-hadoop(greater-than-or-equal-3.1.0)" >}})
Review Comment:
This is a bit strange. in my text editor, it does align with the line above. However, the display here is misaligned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899400719
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
Review Comment:
This is just a directory in our production environment; I'm just illustrating how to install.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899394030
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
Review Comment:
what is the `hmsclient01` and `opt` folder, is that from docker container?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1899394370
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,75 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6 cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-SNAPSHOT-bin
Review Comment:
why use SNAPSHOT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #24:
URL: https://github.com/apache/hive-site/pull/24#discussion_r1900119718
##########
content/docs/latest/manual-installation_283118363.md:
##########
@@ -378,6 +379,77 @@ That directory should contain all the files necessary to run Hive. You can run i
From now, you can follow the steps described in the section Installing Hive from a Tarball
+## Installing with old version hadoop(>=3.1.0)
+
+Although we normally require hive4 to rely on a
+hadoop 3.3.6+ cluster environment.
+However, in practice, in an ON YARN environment,
+we can package all the hadoop related dependencies into
+tez&hive so that they do not need to rely on the lib
+of the original hadoop cluster environment at runtime.
+In this way, we can run HIVE4 in a lower version of hadoop,
+provided that the base APIs of the hadoop 3.x series are common to
+each other.
+
+The steps are as follows:
+
+1.Download the high version of the Hadoop package, unzip it, and then set the hadoop_home finger of the env script in HIVE4 to the path where the high version of hadoop is unzipped.
+
+2.Compile TEZ to get tez.tar.gz which contains all hadoop related dependencies,
+first extract it on the physical machine where HIVE is deployed and configure the TEZ_HOME in HIVE to point to it,
+then place tez.tar.gz in a path in hdfs.
+
+```shell
+## This is an example,Users should install HIVE and TEZ into actual directories.
+## In this example, we have installed HIVE-4.0.1 and TEZ-0.10.4 on an Hadoop 3.1.0 cluster.
+[root@hmsclient01 opt]# cd /opt
+[root@hmsclient01 opt]# ll
+drwxr-xr-x 11 hive hadoop 4096 Nov 7 13:59 apache-hive-4.0.1-bin
+drwxr-xr-x 3 hive hadoop 4096 Nov 7 13:59 apache-tez-0.10.4-bin
+drwxr-xr-x 10 hive hadoop 4096 Nov 7 13:59 hadoop-3.3.6
+lrwxrwxrwx 1 hive hadoop 30 Nov 7 13:59 hive-4.0.0 -> apache-hive-4.0.1-bin
+lrwxrwxrwx 1 hive hadoop 21 Nov 7 13:59 tez -> apache-tez-0.10.4-bin
+```
+
+edit `hive-env.sh`
+
+```shell
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+export TEZ_HOME=/opt/tez
+# Set HADOOP_HOME to point to a specific hadoop install directory
+#HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client}
Review Comment:
Can we remove this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on PR #24:
URL: https://github.com/apache/hive-site/pull/24#issuecomment-2566943147
@zhangbutao @deniskuzZ @ayushtkn
hello sir. I have reformatted the document once, please check it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28683:Add doc for install hive4 with old version hadoop [hive-site]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao merged PR #24:
URL: https://github.com/apache/hive-site/pull/24
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org