You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Fred Bai <pi...@gmail.com> on 2022/08/24 03:45:53 UTC
Should we consider Spark3 support for Hive on Spark
Hi everyone:
Do we have any support for Hive on Spark? I need Hive on Spark, but my
Spark version is 3.X.
I found Hive incompatible with Spark3, I modify a lot of code to be
compatible.
Hive on Spark has deprecated?
And. Hive on Spark is very slow when the job executes.
Re: Should we consider Spark3 support for Hive on Spark
Posted by Sungwoo Park <gl...@gmail.com>.
Thanks for sharing the story. Out of curiosity, I wonder why your client
wants to use Hive on Spark, especially if both Hive 3 and Spark 3 are
running fine. Hive on Spark is hard to install and configure, and does not
run any faster than Hive-Tez/Hive-LLAP and is also much less stable.
(Besides, it will be discontinued from Hive 4.) With Hive 3 and/or Spark 3
installed already, one could just use Hive of Spark SQL.
As a side note, we recently compared the performance of Hive 3 and Spark
3.2.1 using 10TB TPC-DS (as part of evaluating Hive-MR3). When tested with
HDFS in an on-prem cluster, Spark 3 is twice faster than Spark 2, but Hive
3 (Hive-MR3 or Hive-LLAP) still far outperforms Spark 3. The result might
be different on public clouds using S3, but should be enough to convince
people that Hive is still going strong. If interested, see:
https://www.datamonad.com/post/2022-04-01-spark-hive-performance-1.4/
--- Sungwoo
On Thu, Aug 25, 2022 at 12:10 PM Fred Bai <pi...@gmail.com> wrote:
> Thanks all for the reply.
>
> I modify a lot of code of Spark3 and Hive3, and it works. Most of the
> changes focus on interface incompatibility.
>
> I used the TPCDS-10TB to test, 20 times slower than SparkSQL, I switch to
> compatibility version, see:
> https://cwiki.apache.org//confluence/display/Hive/Hive+on+Spark:+Getting+Started
>
> It's still very slow and almost every query has this problem, my client is
> already in use hive on spark2, and then asks to upgrade to spark3.
> [image: image.png]
>
>
> thx.
>
>
> Jan Fili <ja...@gmail.com> 于2022年8月25日周四 01:05写道：
>
>> Yes exactly.
>>
>> This is what is recommended, cause hive on Spark has little interest.
>> However there is nothing enforcing not todo it.
>>
>> Important to me cause i sit here and work on grassroots marrieng hive on
>> kafka-streams.
>>
>> Owen O'Malley <ow...@gmail.com> schrieb am Mi., 24. Aug. 2022,
>> 18:51:
>>
>>> Hive on Spark is not recommended. The recommended path is to use either
>>> Tez or LLAP. If you already are using Spark 3, it would be far easier to
>>> use Spark SQL.
>>>
>>> .. Owen
>>>
>>> On Wed, Aug 24, 2022 at 3:46 AM Fred Bai <pi...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone:
>>>>
>>>> Do we have any support for Hive on Spark? I need Hive on Spark, but my
>>>> Spark version is 3.X.
>>>>
>>>> I found Hive incompatible with Spark3, I modify a lot of code to be
>>>> compatible.
>>>>
>>>> Hive on Spark has deprecated?
>>>>
>>>> And. Hive on Spark is very slow when the job executes.
>>>>
>>>
Re: Should we consider Spark3 support for Hive on Spark
Posted by Fred Bai <pi...@gmail.com>.
Thanks all for the reply.
I modify a lot of code of Spark3 and Hive3, and it works. Most of the
changes focus on interface incompatibility.
I used the TPCDS-10TB to test, 20 times slower than SparkSQL, I switch to
compatibility version, see:
https://cwiki.apache.org//confluence/display/Hive/Hive+on+Spark:+Getting+Started
It's still very slow and almost every query has this problem, my client is
already in use hive on spark2, and then asks to upgrade to spark3.
[image: image.png]
thx.
Jan Fili <ja...@gmail.com> 于2022年8月25日周四 01:05写道：
> Yes exactly.
>
> This is what is recommended, cause hive on Spark has little interest.
> However there is nothing enforcing not todo it.
>
> Important to me cause i sit here and work on grassroots marrieng hive on
> kafka-streams.
>
> Owen O'Malley <ow...@gmail.com> schrieb am Mi., 24. Aug. 2022,
> 18:51:
>
>> Hive on Spark is not recommended. The recommended path is to use either
>> Tez or LLAP. If you already are using Spark 3, it would be far easier to
>> use Spark SQL.
>>
>> .. Owen
>>
>> On Wed, Aug 24, 2022 at 3:46 AM Fred Bai <pi...@gmail.com>
>> wrote:
>>
>>> Hi everyone:
>>>
>>> Do we have any support for Hive on Spark? I need Hive on Spark, but my
>>> Spark version is 3.X.
>>>
>>> I found Hive incompatible with Spark3, I modify a lot of code to be
>>> compatible.
>>>
>>> Hive on Spark has deprecated?
>>>
>>> And. Hive on Spark is very slow when the job executes.
>>>
>>
Re: Should we consider Spark3 support for Hive on Spark
Posted by Jan Fili <ja...@gmail.com>.
Yes exactly.
This is what is recommended, cause hive on Spark has little interest.
However there is nothing enforcing not todo it.
Important to me cause i sit here and work on grassroots marrieng hive on
kafka-streams.
Owen O'Malley <ow...@gmail.com> schrieb am Mi., 24. Aug. 2022, 18:51:
> Hive on Spark is not recommended. The recommended path is to use either
> Tez or LLAP. If you already are using Spark 3, it would be far easier to
> use Spark SQL.
>
> .. Owen
>
> On Wed, Aug 24, 2022 at 3:46 AM Fred Bai <pi...@gmail.com>
> wrote:
>
>> Hi everyone:
>>
>> Do we have any support for Hive on Spark? I need Hive on Spark, but my
>> Spark version is 3.X.
>>
>> I found Hive incompatible with Spark3, I modify a lot of code to be
>> compatible.
>>
>> Hive on Spark has deprecated?
>>
>> And. Hive on Spark is very slow when the job executes.
>>
>
Re: Should we consider Spark3 support for Hive on Spark
Posted by Owen O'Malley <ow...@gmail.com>.
Hive on Spark is not recommended. The recommended path is to use either Tez
or LLAP. If you already are using Spark 3, it would be far easier to use
Spark SQL.
.. Owen
On Wed, Aug 24, 2022 at 3:46 AM Fred Bai <pi...@gmail.com> wrote:
> Hi everyone:
>
> Do we have any support for Hive on Spark? I need Hive on Spark, but my
> Spark version is 3.X.
>
> I found Hive incompatible with Spark3, I modify a lot of code to be
> compatible.
>
> Hive on Spark has deprecated?
>
> And. Hive on Spark is very slow when the job executes.
>
Re: Should we consider Spark3 support for Hive on Spark
Posted by hernan saab via user <us...@hive.apache.org>.
Do you honestly believe that a non apache community dev can just fork hive, modify the code and make it work with any version of spark? Is that what you are suggesting? Please, let us know if this is the case.
Sent from Yahoo Mail for iPad
On Wednesday, August 24, 2022, 6:13 AM, Jan Fili <ja...@gmail.com> wrote:
Can always fork to get things going ;)
*sorry for spam*
Am Mi., 24. Aug. 2022 um 06:34 Uhr schrieb hernan saab via user
<us...@hive.apache.org>:
>
>
> Hey Fred,
>
> Contrary to what you may perceive from the hive docs, what you are trying to do is not plug and play.
> Only apache committers can do what you are trying to do.
> Use canned solutions such as confluence or AWS EMR and save yourself weeks of wasted effort.
>
> Hernán
> On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai <pi...@gmail.com> wrote:
>
>
> Hi everyone:
>
> Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark version is 3.X.
>
> I found Hive incompatible with Spark3, I modify a lot of code to be compatible.
>
> Hive on Spark has deprecated?
>
> And. Hive on Spark is very slow when the job executes.
Re: Should we consider Spark3 support for Hive on Spark
Posted by Jan Fili <ja...@gmail.com>.
Can always fork to get things going ;)
*sorry for spam*
Am Mi., 24. Aug. 2022 um 06:34 Uhr schrieb hernan saab via user
<us...@hive.apache.org>:
>
>
> Hey Fred,
>
> Contrary to what you may perceive from the hive docs, what you are trying to do is not plug and play.
> Only apache committers can do what you are trying to do.
> Use canned solutions such as confluence or AWS EMR and save yourself weeks of wasted effort.
>
> Hernán
> On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai <pi...@gmail.com> wrote:
>
>
> Hi everyone:
>
> Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark version is 3.X.
>
> I found Hive incompatible with Spark3, I modify a lot of code to be compatible.
>
> Hive on Spark has deprecated?
>
> And. Hive on Spark is very slow when the job executes.
Re: Should we consider Spark3 support for Hive on Spark
Posted by hernan saab via user <us...@hive.apache.org>.
Hey Fred,
Contrary to what you may perceive from the hive docs, what you are trying to do is not plug and play.Only apache committers can do what you are trying to do.Use canned solutions such as confluence or AWS EMR and save yourself weeks of wasted effort.
Hernán On Tuesday, August 23, 2022 at 08:46:30 PM PDT, Fred Bai <pi...@gmail.com> wrote:
Hi everyone:
Do we have any support for Hive on Spark? I need Hive on Spark, but my Spark version is 3.X.
I found Hive incompatible with Spark3, I modify a lot of code to be compatible.
Hive on Spark has deprecated?
And. Hive on Spark is very slow when the job executes.
Re: Should we consider Spark3 support for Hive on Spark
Posted by Owen O'Malley <ow...@gmail.com>.
Hive on Spark is not recommended. The recommended path is to use either Tez
or LLAP. If you already are using Spark 3, it would be far easier to use
Spark SQL.
.. Owen
On Wed, Aug 24, 2022 at 3:46 AM Fred Bai <pi...@gmail.com> wrote:
> Hi everyone:
>
> Do we have any support for Hive on Spark? I need Hive on Spark, but my
> Spark version is 3.X.
>
> I found Hive incompatible with Spark3, I modify a lot of code to be
> compatible.
>
> Hive on Spark has deprecated?
>
> And. Hive on Spark is very slow when the job executes.
>