You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "bit1129@163.com" <bi...@163.com> on 2015/03/13 09:24:09 UTC
Explanation on the Hive in the Spark assembly
Hi, sparkers,
I am kind of confused about hive in the spark assembly. I think hive in the spark assembly is not the same thing as Hive On Spark(Hive On Spark, is meant to run hive using spark execution engine).
So, my question is:
1. What is the difference between Hive in the spark assembly and Hive on Hadoop?
2. Does Hive in the spark assembly use Spark execution engine or Hadoop MR engine?
Thanks.
bit1129@163.com
Re: Explanation on the Hive in the Spark assembly
Posted by "bit1129@163.com" <bi...@163.com>.
Can anyone have a look on this question? Thanks.
bit1129@163.com
From: bit1129@163.com
Date: 2015-03-13 16:24
To: user
Subject: Explanation on the Hive in the Spark assembly
Hi, sparkers,
I am kind of confused about hive in the spark assembly. I think hive in the spark assembly is not the same thing as Hive On Spark(Hive On Spark, is meant to run hive using spark execution engine).
So, my question is:
1. What is the difference between Hive in the spark assembly and Hive on Hadoop?
2. Does Hive in the spark assembly use Spark execution engine or Hadoop MR engine?
Thanks.
bit1129@163.com
Re: Re: Explanation on the Hive in the Spark assembly
Posted by "bit1129@163.com" <bi...@163.com>.
Thanks Cheng for the great explanation!
bit1129@163.com
From: Cheng Lian
Date: 2015-03-16 00:53
To: bit1129@163.com; Wang, Daoyuan; user
Subject: Re: Explanation on the Hive in the Spark assembly
Spark SQL supports most commonly used features of HiveQL. However, different HiveQL statements are executed in different manners:
DDL statements (e.g. CREATE TABLE, DROP TABLE, etc.) and commands (e.g. SET <key> = <value>, ADD FILE, ADD JAR, etc.)
In most cases, Spark SQL simply delegates these statements to Hive, as they don’t need to issue any distributed jobs and don’t rely on the computation engine (Spark, MR, or Tez).
SELECT queries, CREATE TABLE ... AS SELECT ... statements and insertions
These statements are executed using Spark as the execution engine.
The Hive classes packaged in the assembly jar are used to provide entry points to Hive features, for example:
HiveQL parser
Talking to Hive metastore to execute DDL statements
Accessing UDF/UDAF/UDTF
As for the differences between Hive on Spark and Spark SQL’s Hive support, please refer to this article by Reynold: https://databricks.com/blog/2014/07/01/shark-spark-sql-hive-on-spark-and-the-future-of-sql-on-spark.html
Cheng
On 3/14/15 10:53 AM, bit1129@163.com wrote:
Thanks Daoyuan.
What do you mean by running some native command, I never thought that hive will run without an computing engine like Hadoop MR or spark. Thanks.
bit1129@163.com
From: Wang, Daoyuan
Date: 2015-03-13 16:39
To: bit1129@163.com; user
Subject: RE: Explanation on the Hive in the Spark assembly
Hi bit1129,
1, hive in spark assembly removed most dependencies of hive on Hadoop to avoid conflicts.
2, this hive is used to run some native command, which does not rely on spark or mapreduce.
Thanks,
Daoyuan
From: bit1129@163.com [mailto:bit1129@163.com]
Sent: Friday, March 13, 2015 4:24 PM
To: user
Subject: Explanation on the Hive in the Spark assembly
Hi, sparkers,
I am kind of confused about hive in the spark assembly. I think hive in the spark assembly is not the same thing as Hive On Spark(Hive On Spark, is meant to run hive using spark execution engine).
So, my question is:
1. What is the difference between Hive in the spark assembly and Hive on Hadoop?
2. Does Hive in the spark assembly use Spark execution engine or Hadoop MR engine?
Thanks.
bit1129@163.com
Re: Explanation on the Hive in the Spark assembly
Posted by Cheng Lian <li...@gmail.com>.
Spark SQL supports most commonly used features of HiveQL. However,
different HiveQL statements are executed in different manners:
1.
DDL statements (e.g. |CREATE TABLE|, |DROP TABLE|, etc.) and
commands (e.g. |SET <key> = <value>|, |ADD FILE|, |ADD JAR|, etc.)
In most cases, Spark SQL simply delegates these statements to Hive,
as they don’t need to issue any distributed jobs and don’t rely on
the computation engine (Spark, MR, or Tez).
2.
|SELECT| queries, |CREATE TABLE ... AS SELECT ...| statements and
insertions
These statements are executed using Spark as the execution engine.
The Hive classes packaged in the assembly jar are used to provide entry
points to Hive features, for example:
1. HiveQL parser
2. Talking to Hive metastore to execute DDL statements
3. Accessing UDF/UDAF/UDTF
As for the differences between Hive on Spark and Spark SQL’s Hive
support, please refer to this article by Reynold:
https://databricks.com/blog/2014/07/01/shark-spark-sql-hive-on-spark-and-the-future-of-sql-on-spark.html
Cheng
On 3/14/15 10:53 AM, bit1129@163.com wrote:
> Thanks Daoyuan.
> What do you mean by running some native command, I never thought that
> hive will run without an computing engine like Hadoop MR or spark. Thanks.
>
> ------------------------------------------------------------------------
> bit1129@163.com
>
> *From:* Wang, Daoyuan <ma...@intel.com>
> *Date:* 2015-03-13 16:39
> *To:* bit1129@163.com <ma...@163.com>; user
> <ma...@spark.apache.org>
> *Subject:* RE: Explanation on the Hive in the Spark assembly
>
> Hi bit1129,
>
> 1, hive in spark assembly removed most dependencies of hive on
> Hadoop to avoid conflicts.
>
> 2, this hive is used to run some native command, which does not
> rely on spark or mapreduce.
>
> Thanks,
>
> Daoyuan
>
> *From:*bit1129@163.com [mailto:bit1129@163.com]
> *Sent:* Friday, March 13, 2015 4:24 PM
> *To:* user
> *Subject:* Explanation on the Hive in the Spark assembly
>
> Hi, sparkers,
>
> I am kind of confused about hive in the spark assembly. I think
> hive in the spark assembly is not the same thing as Hive On
> Spark(Hive On Spark, is meant to run hive using spark execution
> engine).
>
> So, my question is:
>
> 1. What is the difference between Hive in the spark assembly and
> Hive on Hadoop?
>
> 2. Does Hive in the spark assembly use Spark execution engine or
> Hadoop MR engine?
>
> Thanks.
>
> ------------------------------------------------------------------------
>
> bit1129@163.com <ma...@163.com>
>
Re: RE: Explanation on the Hive in the Spark assembly
Posted by "bit1129@163.com" <bi...@163.com>.
Thanks Daoyuan.
What do you mean by running some native command, I never thought that hive will run without an computing engine like Hadoop MR or spark. Thanks.
bit1129@163.com
From: Wang, Daoyuan
Date: 2015-03-13 16:39
To: bit1129@163.com; user
Subject: RE: Explanation on the Hive in the Spark assembly
Hi bit1129,
1, hive in spark assembly removed most dependencies of hive on Hadoop to avoid conflicts.
2, this hive is used to run some native command, which does not rely on spark or mapreduce.
Thanks,
Daoyuan
From: bit1129@163.com [mailto:bit1129@163.com]
Sent: Friday, March 13, 2015 4:24 PM
To: user
Subject: Explanation on the Hive in the Spark assembly
Hi, sparkers,
I am kind of confused about hive in the spark assembly. I think hive in the spark assembly is not the same thing as Hive On Spark(Hive On Spark, is meant to run hive using spark execution engine).
So, my question is:
1. What is the difference between Hive in the spark assembly and Hive on Hadoop?
2. Does Hive in the spark assembly use Spark execution engine or Hadoop MR engine?
Thanks.
bit1129@163.com
RE: Explanation on the Hive in the Spark assembly
Posted by "Wang, Daoyuan" <da...@intel.com>.
Hi bit1129,
1, hive in spark assembly removed most dependencies of hive on Hadoop to avoid conflicts.
2, this hive is used to run some native command, which does not rely on spark or mapreduce.
Thanks,
Daoyuan
From: bit1129@163.com [mailto:bit1129@163.com]
Sent: Friday, March 13, 2015 4:24 PM
To: user
Subject: Explanation on the Hive in the Spark assembly
Hi, sparkers,
I am kind of confused about hive in the spark assembly. I think hive in the spark assembly is not the same thing as Hive On Spark(Hive On Spark, is meant to run hive using spark execution engine).
So, my question is:
1. What is the difference between Hive in the spark assembly and Hive on Hadoop?
2. Does Hive in the spark assembly use Spark execution engine or Hadoop MR engine?
Thanks.
________________________________
bit1129@163.com<ma...@163.com>