You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/03/26 22:47:20 UTC
Hive on Spark engine
Hi,
I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
that Hive can utilise a Spark engine higher than 1.3.1
My understanding was that there were miss-match on Hadoop assembly Jar
files that cause Hive not being able to run on Spark using the binary
downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
crashed.
I do not know the development state of this cross-breed but will be very
desirable if we could manage to sort out
this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
Re: Hive on Spark engine
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Ted,
More interested in general availability of Hive 2 on Spark 1.6 engine as
opposed to Vendors specific custom built.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 26 March 2016 at 23:55, Ted Yu <yu...@gmail.com> wrote:
> According to:
>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf
>
> Spark 1.5.2 comes out of box.
>
> Suggest moving questions on HDP to Hortonworks forum.
>
> Cheers
>
> On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Thanks Jorn.
>>
>> Just to be clear they get Hive working with Spark 1.6 out of the box
>> (binary download)? The usual work-around is to build your own package and
>> get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
>>
>>
>> Cheers
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
>>
>>> If you check the newest Hortonworks distribution then you see that it
>>> generally works. Maybe you can borrow some of their packages. Alternatively
>>> it should be also available in other distributions.
>>>
>>> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
>>> that Hive can utilise a Spark engine higher than 1.3.1
>>>
>>> My understanding was that there were miss-match on Hadoop assembly Jar
>>> files that cause Hive not being able to run on Spark using the binary
>>> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
>>> crashed.
>>>
>>> I do not know the development state of this cross-breed but will be very
>>> desirable if we could manage to sort out
>>> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>>>
>>> Thanks
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>>
>>
>
Re: Hive on Spark engine
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Ted,
More interested in general availability of Hive 2 on Spark 1.6 engine as
opposed to Vendors specific custom built.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 26 March 2016 at 23:55, Ted Yu <yu...@gmail.com> wrote:
> According to:
>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf
>
> Spark 1.5.2 comes out of box.
>
> Suggest moving questions on HDP to Hortonworks forum.
>
> Cheers
>
> On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Thanks Jorn.
>>
>> Just to be clear they get Hive working with Spark 1.6 out of the box
>> (binary download)? The usual work-around is to build your own package and
>> get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
>>
>>
>> Cheers
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
>>
>>> If you check the newest Hortonworks distribution then you see that it
>>> generally works. Maybe you can borrow some of their packages. Alternatively
>>> it should be also available in other distributions.
>>>
>>> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
>>> that Hive can utilise a Spark engine higher than 1.3.1
>>>
>>> My understanding was that there were miss-match on Hadoop assembly Jar
>>> files that cause Hive not being able to run on Spark using the binary
>>> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
>>> crashed.
>>>
>>> I do not know the development state of this cross-breed but will be very
>>> desirable if we could manage to sort out
>>> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>>>
>>> Thanks
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>>
>>
>
Re: Hive on Spark engine
Posted by Ted Yu <yu...@gmail.com>.
According to:
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf
Spark 1.5.2 comes out of box.
Suggest moving questions on HDP to Hortonworks forum.
Cheers
On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Jorn.
>
> Just to be clear they get Hive working with Spark 1.6 out of the box
> (binary download)? The usual work-around is to build your own package and
> get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
>
>
> Cheers
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
>
>> If you check the newest Hortonworks distribution then you see that it
>> generally works. Maybe you can borrow some of their packages. Alternatively
>> it should be also available in other distributions.
>>
>> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
>> that Hive can utilise a Spark engine higher than 1.3.1
>>
>> My understanding was that there were miss-match on Hadoop assembly Jar
>> files that cause Hive not being able to run on Spark using the binary
>> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
>> crashed.
>>
>> I do not know the development state of this cross-breed but will be very
>> desirable if we could manage to sort out
>> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>>
>> Thanks
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>>
>
Re: Hive on Spark engine
Posted by Ted Yu <yu...@gmail.com>.
According to:
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/bk_HDP_RelNotes-20151221.pdf
Spark 1.5.2 comes out of box.
Suggest moving questions on HDP to Hortonworks forum.
Cheers
On Sat, Mar 26, 2016 at 3:32 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Jorn.
>
> Just to be clear they get Hive working with Spark 1.6 out of the box
> (binary download)? The usual work-around is to build your own package and
> get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
>
>
> Cheers
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
>
>> If you check the newest Hortonworks distribution then you see that it
>> generally works. Maybe you can borrow some of their packages. Alternatively
>> it should be also available in other distributions.
>>
>> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
>> that Hive can utilise a Spark engine higher than 1.3.1
>>
>> My understanding was that there were miss-match on Hadoop assembly Jar
>> files that cause Hive not being able to run on Spark using the binary
>> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
>> crashed.
>>
>> I do not know the development state of this cross-breed but will be very
>> desirable if we could manage to sort out
>> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>>
>> Thanks
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>>
>
Re: Hive on Spark engine
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Jorn.
Just to be clear they get Hive working with Spark 1.6 out of the box
(binary download)? The usual work-around is to build your own package and
get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
Cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
> If you check the newest Hortonworks distribution then you see that it
> generally works. Maybe you can borrow some of their packages. Alternatively
> it should be also available in other distributions.
>
> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Hi,
>
> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
> that Hive can utilise a Spark engine higher than 1.3.1
>
> My understanding was that there were miss-match on Hadoop assembly Jar
> files that cause Hive not being able to run on Spark using the binary
> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
> crashed.
>
> I do not know the development state of this cross-breed but will be very
> desirable if we could manage to sort out
> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
Re: Hive on Spark engine
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Jorn.
Just to be clear they get Hive working with Spark 1.6 out of the box
(binary download)? The usual work-around is to build your own package and
get the Hadoop-assembly jar file copied over to $HIVE_HOME/lib.
Cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 26 March 2016 at 22:08, Jörn Franke <jo...@gmail.com> wrote:
> If you check the newest Hortonworks distribution then you see that it
> generally works. Maybe you can borrow some of their packages. Alternatively
> it should be also available in other distributions.
>
> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Hi,
>
> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign
> that Hive can utilise a Spark engine higher than 1.3.1
>
> My understanding was that there were miss-match on Hadoop assembly Jar
> files that cause Hive not being able to run on Spark using the binary
> downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it
> crashed.
>
> I do not know the development state of this cross-breed but will be very
> desirable if we could manage to sort out
> this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
Re: Hive on Spark engine
Posted by Jörn Franke <jo...@gmail.com>.
If you check the newest Hortonworks distribution then you see that it generally works. Maybe you can borrow some of their packages. Alternatively it should be also available in other distributions.
> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Hi,
>
> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign that Hive can utilise a Spark engine higher than 1.3.1
>
> My understanding was that there were miss-match on Hadoop assembly Jar files that cause Hive not being able to run on Spark using the binary downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it crashed.
>
> I do not know the development state of this cross-breed but will be very desirable if we could manage to sort out this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>
Re: Hive on Spark engine
Posted by Jörn Franke <jo...@gmail.com>.
If you check the newest Hortonworks distribution then you see that it generally works. Maybe you can borrow some of their packages. Alternatively it should be also available in other distributions.
> On 26 Mar 2016, at 22:47, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Hi,
>
> I am running Hive 2 and now Spark 1.6.1 but I still do not see any sign that Hive can utilise a Spark engine higher than 1.3.1
>
> My understanding was that there were miss-match on Hadoop assembly Jar files that cause Hive not being able to run on Spark using the binary downloads. I just tried Hive 2 on Spark 1.6 as the execution engine and it crashed.
>
> I do not know the development state of this cross-breed but will be very desirable if we could manage to sort out this spark-assembly-1.x.1-hadoop2.4.0.jar for once.
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>