What has changed in Hive 2.3.2 that cannot use Spark engine.

[Mich Talebzadeh <mi...@gmail.com>]

Hi,

my Hive 2.0.1 works fine on Spark 1.3.1 engine.

select count(1) from sales;
Starting Spark Job = dc529d0e-e2d2-431f-8c17-f7867858217f
Query Hive on Spark job[5] stages:
10
11
Status: Running (Hive on Spark job[5])

However, once I upgraded Hive to version 2.3.2, it stopped working!

Any ideas what has changed?

Thanks


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: What has changed in Hive 2.3.2 that cannot use Spark engine.

[Sahil Takiar <ta...@gmail.com>]

Hive 2.0.1 only supports Spark 1.5.0, Hive 2.3.2 only supports Spark 2.0.0

There was probably some backwards incompatible change in Spark from 1.x to
2.x that made it stop working.

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-VersionCompatibility

If there is interest in running newer version of Hive with older versions
of Spark we can consider adding a Spark shims layer to Hive, similar to
what we do to support multiple Hadoop versions.

On Fri, May 4, 2018 at 1:24 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:

> Hi,
>
> my Hive 2.0.1 works fine on Spark 1.3.1 engine.
>
> select count(1) from sales;
> Starting Spark Job = dc529d0e-e2d2-431f-8c17-f7867858217f
> Query Hive on Spark job[5] stages:
> 10
> 11
> Status: Running (Hive on Spark job[5])
>
> However, once I upgraded Hive to version 2.3.2, it stopped working!
>
> Any ideas what has changed?
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>



-- 
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309