You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Raajay <ra...@gmail.com> on 2015/08/07 22:06:18 UTC

Running hive on tez locally

I have been running Hive queries on a single node (no HDFS). I realize that
the queries get compiled as map-reduce jobs and not as TEZ jobs even though
"hive.execution.engine=tez" is set.

Is that expected ? If yes, what is the ideal environment for debugging hive
on tez?

Raajay

Re: Running hive on tez locally

Posted by Prasanth Jayachandran <pj...@hortonworks.com>.

Hi

Can you make sure the following configs are set and appropriately pointing to your corresponding local directories?

set hive.user.install.directory=file:///tmp;
set fs.default.name=file:///;
set fs.defaultFS=file:///;
set tez.staging-dir=/tmp;
set tez.ignore.lib.uris=true;
set tez.runtime.optimize.local.fetch=true;
set tez.local.mode=true;
set hive.execution.engine=tez;

Thanks
Prasanth

> On Aug 7, 2015, at 1:06 PM, Raajay <ra...@gmail.com> wrote:
> 
> I have been running Hive queries on a single node (no HDFS). I realize that the queries get compiled as map-reduce jobs and not as TEZ jobs even though "hive.execution.engine=tez" is set.
> 
> Is that expected ? If yes, what is the ideal environment for debugging hive on tez?
> 
> Raajay

Re: Running hive on tez locally

Posted by Jason Dere <jd...@hortonworks.com>.

I have my HADOOP_CLASSPATH set to point to both $TEZ_HOME/* and $TEZ_HOME/lib/*. Also have TEZ_H0ME and TEZ_CONF_DIR set, though I'm not sure if these are required or not.?

________________________________
From: Raajay <ra...@gmail.com>
Sent: Friday, August 07, 2015 2:50 PM
To: user@hive.apache.org
Subject: Re: Running hive on tez locally

Thanks for the configs. When I run hive it crashes because TEZ libraries were not found.

How do I point Hive to tez libraries? Is it sufficient to populate CLASSPATH environment variables with location of tez libraries ?

Raajay

On Fri, Aug 7, 2015 at 3:16 PM, Jason Dere <jd...@hortonworks.com>> wrote:

If you add these settings (paths are up to you), you can run Hive with Tez in local mode. You'll also need to make sure your classpath has the Tez libs etc.

set hive.user.install.directory=file:///tmp;
set fs.default.name<http://fs.default.name>=file:///;
set fs.defaultFS=file:///;
set tez.staging-dir=/tmp;
set tez.ignore.lib.uris=true;
set tez.runtime.optimize.local.fetch=true;
set tez.local.mode=true;
set hive.execution.engine=tez;?

?

________________________________
From: Raajay <ra...@gmail.com>>
Sent: Friday, August 07, 2015 1:06 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Running hive on tez locally

I have been running Hive queries on a single node (no HDFS). I realize that the queries get compiled as map-reduce jobs and not as TEZ jobs even though "hive.execution.engine=tez" is set.

Is that expected ? If yes, what is the ideal environment for debugging hive on tez?

Raajay

Re: Running hive on tez locally

Posted by Gopal Vijayaraghavan <go...@apache.org>.

> How do I point Hive to tez libraries? Is it sufficient to populate
>CLASSPATH environment variables with location of tez libraries ?

The easiest option there is to populate the hive-config.sh with these two
parameters

https://github.com/t3rmin4t0r/tez-autobuild/blob/master/Makefile#L138


Once you do that, the bin/hive will always load the Tez jars correctly.

Cheers,
Gopal

Re: Running hive on tez locally

Posted by Raajay <ra...@gmail.com>.

Thanks for the configs. When I run hive it crashes because TEZ libraries
were not found.

How do I point Hive to tez libraries? Is it sufficient to populate
CLASSPATH environment variables with location of tez libraries ?

Raajay



On Fri, Aug 7, 2015 at 3:16 PM, Jason Dere <jd...@hortonworks.com> wrote:

> If you add these settings (paths are up to you), you can run Hive with Tez
> in local mode. You'll also need to make sure your classpath has the Tez
> libs etc.
>
>
> set hive.user.install.directory=file:///tmp;
> set fs.default.name=file:///;
> set fs.defaultFS=file:///;
> set tez.staging-dir=/tmp;
> set tez.ignore.lib.uris=true;
> set tez.runtime.optimize.local.fetch=true;
> set tez.local.mode=true;
> set hive.execution.engine=tez;
>
>
> 
>
>
>
> ------------------------------
> *From:* Raajay <ra...@gmail.com>
> *Sent:* Friday, August 07, 2015 1:06 PM
> *To:* user@hive.apache.org
> *Subject:* Running hive on tez locally
>
> I have been running Hive queries on a single node (no HDFS). I realize
> that the queries get compiled as map-reduce jobs and not as TEZ jobs even
> though "hive.execution.engine=tez" is set.
>
> Is that expected ? If yes, what is the ideal environment for debugging
> hive on tez?
>
> Raajay
>

Re: Running hive on tez locally

Posted by Jason Dere <jd...@hortonworks.com>.

If you add these settings (paths are up to you), you can run Hive with Tez in local mode. You'll also need to make sure your classpath has the Tez libs etc.


set hive.user.install.directory=file:///tmp;
set fs.default.name=file:///;
set fs.defaultFS=file:///;
set tez.staging-dir=/tmp;
set tez.ignore.lib.uris=true;
set tez.runtime.optimize.local.fetch=true;
set tez.local.mode=true;
set hive.execution.engine=tez;?


?



________________________________
From: Raajay <ra...@gmail.com>
Sent: Friday, August 07, 2015 1:06 PM
To: user@hive.apache.org
Subject: Running hive on tez locally

I have been running Hive queries on a single node (no HDFS). I realize that the queries get compiled as map-reduce jobs and not as TEZ jobs even though "hive.execution.engine=tez" is set.

Is that expected ? If yes, what is the ideal environment for debugging hive on tez?

Raajay