You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/05/29 20:26:05 UTC
Anyone successfully deployed Hive on TEZ engine?
Please bear in mind that I am talking about your own build not anything
comes as part of Vendor's package.
If so kindly specify both Hive and TEZ versions.
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Gopal,
please see my correspondence about Tez in tez user group. I forwarded to
hive user group.
thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 30 May 2016 at 12:30, Gopal Vijayaraghavan <go...@apache.org> wrote:
> > I do not use any vendor's product., All my own set up, build and
> >configure.
>
> My autobuild scripts should serve as readable documentation for this,
> since nearly everything's in a single Makefile with an install: target.
>
> Or take the easy route with
>
> $ make dist install
>
> In case you use the llap branch, just do "set
> hive.llap.execution.mode=none;" to use Tez.
>
> > java version "1.8.0_77"
> > Hadoop 2.6.0
> ...
> > https://tez.apache.org/install.html
>
> Looks good so far.
>
> > Ok I just need to make it work as I have hive on spark engine as well.
>
> You're missing 3 things approximately - if you read through the Makefile
> in github.
>
> First, a good tez-site.xml in the classpath (remember, tez.lib.uris needs
> to be an HDFS path - for the rest, see the base file from autobuild).
>
> I usually update Tez to ${fs.default.name}/user/gopal/tez/tez.tar.gz and I
> do not use the minimal tarball, but the full dist tarball.
>
> The fixed tarball means it hits all the good localization characteristics
> of YARN, which can add up to minutes on a >250+ node cluster.
>
> Second, put that in the classpath for Hive (append to
> $INSTALL_ROOT/hive/bin/hive-config.sh)
>
> > export
> >HADOOP_CLASSPATH="$INSTALL_ROOT/tez/*:$INSTALL_ROOT/tez/lib/*:$INSTALL_ROO
> >T/tez/conf/:$HADOOP_CLASSPATH"
>
> > export HADOOP_USER_CLASSPATH_FIRST=true
>
>
> Replace $INSTALL_ROOT with wherever Tez is located.
>
> Third, disable the hive-1.x jars coming from SparkSQL (append/create in
> $INSTALL_ROOT/hive/conf/hive-env.sh)
>
> > export HIVE_SKIP_SPARK_ASSEMBLY=true
>
>
> After that, you can do
>
> > hive --hiveconf hive.execution.engine=tez
>
> to get Tez working (add --hiveconf tez.queue.name=<queue> to use queues).
>
> Cheers,
> Gopal
>
>
>
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> I do not use any vendor's product., All my own set up, build and
>configure.
My autobuild scripts should serve as readable documentation for this,
since nearly everything's in a single Makefile with an install: target.
Or take the easy route with
$ make dist install
In case you use the llap branch, just do "set
hive.llap.execution.mode=none;" to use Tez.
> java version "1.8.0_77"
> Hadoop 2.6.0
...
> https://tez.apache.org/install.html
Looks good so far.
> Ok I just need to make it work as I have hive on spark engine as well.
You're missing 3 things approximately - if you read through the Makefile
in github.
First, a good tez-site.xml in the classpath (remember, tez.lib.uris needs
to be an HDFS path - for the rest, see the base file from autobuild).
I usually update Tez to ${fs.default.name}/user/gopal/tez/tez.tar.gz and I
do not use the minimal tarball, but the full dist tarball.
The fixed tarball means it hits all the good localization characteristics
of YARN, which can add up to minutes on a >250+ node cluster.
Second, put that in the classpath for Hive (append to
$INSTALL_ROOT/hive/bin/hive-config.sh)
> export
>HADOOP_CLASSPATH="$INSTALL_ROOT/tez/*:$INSTALL_ROOT/tez/lib/*:$INSTALL_ROO
>T/tez/conf/:$HADOOP_CLASSPATH"
> export HADOOP_USER_CLASSPATH_FIRST=true
Replace $INSTALL_ROOT with wherever Tez is located.
Third, disable the hive-1.x jars coming from SparkSQL (append/create in
$INSTALL_ROOT/hive/conf/hive-env.sh)
> export HIVE_SKIP_SPARK_ASSEMBLY=true
After that, you can do
> hive --hiveconf hive.execution.engine=tez
to get Tez working (add --hiveconf tez.queue.name=<queue> to use queues).
Cheers,
Gopal
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Gopal.
I do not use any vendor's product., All my own set up, build and configure.
No CDH no HDL etc.
This the current stack that I have:
Java
*java -version*
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
HDFS version
*hadoop version*Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
YARN version
*yarn version*Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
HIVE version
hive --version
Hive 2.0.0
Subversion git://reznor-mbp-2.local/Users/sergey/git/hivegit -r
7f9f1fcb8697fb33f0edc2c391930a3728d247d7
Compiled by sergey on Tue Feb 9 18:12:08 PST 2016
Spark version
version 1.6.1
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_77)
TEZ I downloaded
tez-0.8.3
And built it using the following instructions
https://tez.apache.org/install.html
Ok I just need to make it work as I have hive on spark engine as well.
please tell me what version of tez and yarn etc. I
thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 30 May 2016 at 10:16, Gopal Vijayaraghavan <go...@apache.org> wrote:
>
> > In short at the simplest set up what Resource Manager it works with?
>
> Tez+Hive needs HDFS and YARN 2.6.0+ (preferably as close to an Apache
> build as possible - CDH clusters need more work).
>
> Hive2 needs Apache Slider 0.91 right now, to start the cache daemons on
> YARN (see SLIDER-82).
>
> > If so kindly specify both Hive and TEZ versions.
>
> I maintain build scripts & configuration setups for Hive+Tez, for work
>
> https://github.com/t3rmin4t0r/tez-autobuild/tree/llap
>
>
> Both that & the master there builds Hive (2.1.0-SNAPSHOT) + Tez
> (0.8.4-SNAPSHOT), this one has the
> LLAP cache configurations turned on.
>
> This is what I use to develop Hive, before there are releases and it will
> allow each user
> on a shared cluster to maintain their own independent private install of
> hive - if you look at
> something like the old Spotify Hive query presentations, you'll see that
> more people have
> used that to run their own private builds successfully :)
>
> Purely out of laziness, the LLAP configurations in slider-gen.sh (i.e the
> Xmx & cache values)
> are configured exactly to match my dev cluster - 32 vcore + 256Gb RAM.
>
> Cheers,
> Gopal
>
>
>
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> In short at the simplest set up what Resource Manager it works with?
Tez+Hive needs HDFS and YARN 2.6.0+ (preferably as close to an Apache
build as possible - CDH clusters need more work).
Hive2 needs Apache Slider 0.91 right now, to start the cache daemons on
YARN (see SLIDER-82).
> If so kindly specify both Hive and TEZ versions.
I maintain build scripts & configuration setups for Hive+Tez, for work
https://github.com/t3rmin4t0r/tez-autobuild/tree/llap
Both that & the master there builds Hive (2.1.0-SNAPSHOT) + Tez
(0.8.4-SNAPSHOT), this one has the
LLAP cache configurations turned on.
This is what I use to develop Hive, before there are releases and it will
allow each user
on a shared cluster to maintain their own independent private install of
hive - if you look at
something like the old Spotify Hive query presentations, you'll see that
more people have
used that to run their own private builds successfully :)
Purely out of laziness, the LLAP configurations in slider-gen.sh (i.e the
Xmx & cache values)
are configured exactly to match my dev cluster - 32 vcore + 256Gb RAM.
Cheers,
Gopal
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Mich Talebzadeh <mi...@gmail.com>.
thanks Damien.
I tried TEZ 0.82 with Hive 2 although I did not persevere.
When you say "Not stable" are you referring to using it with YARN etc.
In short at the simplest set up what Resource Manager it works with?
Cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 30 May 2016 at 08:59, Damien Carol <da...@gmail.com> wrote:
> HIVE 1.2.1 and Tez 0.5.2 or 0.7.0 works pretty well
>
> beginning to use HIVE 2.0.0 and 0.8.x but not stable :/
>
> 2016-05-29 22:26 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>
>>
>> Please bear in mind that I am talking about your own build not anything
>> comes as part of Vendor's package.
>>
>> If so kindly specify both Hive and TEZ versions.
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
Re: Anyone successfully deployed Hive on TEZ engine?
Posted by Damien Carol <da...@gmail.com>.
HIVE 1.2.1 and Tez 0.5.2 or 0.7.0 works pretty well
beginning to use HIVE 2.0.0 and 0.8.x but not stable :/
2016-05-29 22:26 GMT+02:00 Mich Talebzadeh <mi...@gmail.com>:
>
> Please bear in mind that I am talking about your own build not anything
> comes as part of Vendor's package.
>
> If so kindly specify both Hive and TEZ versions.
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>