You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Kali.tummala@gmail.com" <Ka...@gmail.com> on 2016/05/20 23:57:00 UTC
set spark 1.6 with Hive 0.14 ?
Hi All ,
Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
of inbuilt hive 1.2.1.
I am testing spark-sql locally by downloading spark 1.6 from internet , I
want to execute my hive queries in spark sql using hive version 0.14 can I
go back to previous version just for a simple test.
Please share out the steps involved.
Thanks
Sri
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: set spark 1.6 with Hive 0.14 ?
Posted by Jörn Franke <jo...@gmail.com>.
What is the motivation to use such an old version of Hive? This will lead to less performance and other risks.
> On 21 May 2016, at 01:57, "Kali.tummala@gmail.com" <Ka...@gmail.com> wrote:
>
> Hi All ,
>
> Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
> of inbuilt hive 1.2.1.
>
> I am testing spark-sql locally by downloading spark 1.6 from internet , I
> want to execute my hive queries in spark sql using hive version 0.14 can I
> go back to previous version just for a simple test.
>
> Please share out the steps involved.
>
>
> Thanks
> Sri
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: set spark 1.6 with Hive 0.14 ?
Posted by Sri <ka...@gmail.com>.
Thanks Ted, I know in spark-she'll can we set same in spark-sql shell ?
If I don't set hive context from my understanding spark is using its own SQL and date functions right ? Like for example interval ?
Thanks
Sri
Sent from my iPhone
> On 21 May 2016, at 08:19, Ted Yu <yu...@gmail.com> wrote:
>
> In spark-shell:
>
> scala> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.hive.HiveContext
>
> scala> var hc: HiveContext = new HiveContext(sc)
>
> FYI
>
>> On Sat, May 21, 2016 at 8:11 AM, Sri <ka...@gmail.com> wrote:
>> Hi ,
>>
>> You mean hive-site.xml file right ?,I did placed the hive-site.xml in spark conf but not sure how spark certain date functions like interval is still working .
>> Hive 0.14 don't have interval function but how spark is managing to do that ?
>> Does spark has its own date functions ? I am using spark-sql shell for your information.
>>
>> Can I set hive context.sql in spark-Sql shell ? As we do in traditional spark Scala application.
>>
>> Thanks
>> Sri
>>
>> Sent from my iPhone
>>
>>> On 21 May 2016, at 02:24, Mich Talebzadeh <mi...@gmail.com> wrote:
>>>
>>> Sou want to use hive version 0.14 when using Spark 1.6?
>>>
>>> Go to directory $SPARK_HOME/conf and create a softlink to hive-core.xml file
>>>
>>> cd $SPARK_HOME
>>> hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6> cd conf
>>> hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6/conf> ls -ltr
>>>
>>> lrwxrwxrwx 1 hduser hadoop 32 May 3 17:48 hive-site.xml -> /usr/lib/hive/conf/hive-site.xml
>>> -
>>>
>>> You can see the softlink in mine. Just create one as below
>>>
>>> ln -s /usr/lib/hive/conf/hive-site.xml hive-site.xml
>>>
>>>
>>> That should work
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>> On 21 May 2016 at 00:57, Kali.tummala@gmail.com <Ka...@gmail.com> wrote:
>>>> Hi All ,
>>>>
>>>> Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
>>>> of inbuilt hive 1.2.1.
>>>>
>>>> I am testing spark-sql locally by downloading spark 1.6 from internet , I
>>>> want to execute my hive queries in spark sql using hive version 0.14 can I
>>>> go back to previous version just for a simple test.
>>>>
>>>> Please share out the steps involved.
>>>>
>>>>
>>>> Thanks
>>>> Sri
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>
>
Re: set spark 1.6 with Hive 0.14 ?
Posted by Ted Yu <yu...@gmail.com>.
In spark-shell:
scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext
scala> var hc: HiveContext = new HiveContext(sc)
FYI
On Sat, May 21, 2016 at 8:11 AM, Sri <ka...@gmail.com> wrote:
> Hi ,
>
> You mean hive-site.xml file right ?,I did placed the hive-site.xml in
> spark conf but not sure how spark certain date functions like interval is
> still working .
> Hive 0.14 don't have interval function but how spark is managing to do
> that ?
> Does spark has its own date functions ? I am using spark-sql shell for
> your information.
>
> Can I set hive context.sql in spark-Sql shell ? As we do in traditional
> spark Scala application.
>
> Thanks
> Sri
>
> Sent from my iPhone
>
> On 21 May 2016, at 02:24, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Sou want to use hive version 0.14 when using Spark 1.6?
>
> Go to directory $SPARK_HOME/conf and create a softlink to hive-core.xml
> file
>
> *cd $SPARK_HOME*
> hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6>
> *cd conf*hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6/conf> ls -ltr
>
> lrwxrwxrwx 1 hduser hadoop 32 May 3 17:48 hive-site.xml ->
> /usr/lib/hive/conf/hive-site.xml
> -
>
> You can see the softlink in mine. Just create one as below
>
> ln -s /usr/lib/hive/conf/hive-site.xml hive-site.xml
>
>
> That should work
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 00:57, Kali.tummala@gmail.com <Ka...@gmail.com>
> wrote:
>
>> Hi All ,
>>
>> Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
>> of inbuilt hive 1.2.1.
>>
>> I am testing spark-sql locally by downloading spark 1.6 from internet , I
>> want to execute my hive queries in spark sql using hive version 0.14 can I
>> go back to previous version just for a simple test.
>>
>> Please share out the steps involved.
>>
>>
>> Thanks
>> Sri
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com
>> <http://nabble.com>.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: set spark 1.6 with Hive 0.14 ?
Posted by Sri <ka...@gmail.com>.
Hi ,
You mean hive-site.xml file right ?,I did placed the hive-site.xml in spark conf but not sure how spark certain date functions like interval is still working .
Hive 0.14 don't have interval function but how spark is managing to do that ?
Does spark has its own date functions ? I am using spark-sql shell for your information.
Can I set hive context.sql in spark-Sql shell ? As we do in traditional spark Scala application.
Thanks
Sri
Sent from my iPhone
> On 21 May 2016, at 02:24, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Sou want to use hive version 0.14 when using Spark 1.6?
>
> Go to directory $SPARK_HOME/conf and create a softlink to hive-core.xml file
>
> cd $SPARK_HOME
> hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6> cd conf
> hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6/conf> ls -ltr
>
> lrwxrwxrwx 1 hduser hadoop 32 May 3 17:48 hive-site.xml -> /usr/lib/hive/conf/hive-site.xml
> -
>
> You can see the softlink in mine. Just create one as below
>
> ln -s /usr/lib/hive/conf/hive-site.xml hive-site.xml
>
>
> That should work
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>
>
>> On 21 May 2016 at 00:57, Kali.tummala@gmail.com <Ka...@gmail.com> wrote:
>> Hi All ,
>>
>> Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
>> of inbuilt hive 1.2.1.
>>
>> I am testing spark-sql locally by downloading spark 1.6 from internet , I
>> want to execute my hive queries in spark sql using hive version 0.14 can I
>> go back to previous version just for a simple test.
>>
>> Please share out the steps involved.
>>
>>
>> Thanks
>> Sri
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
Re: set spark 1.6 with Hive 0.14 ?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Sou want to use hive version 0.14 when using Spark 1.6?
Go to directory $SPARK_HOME/conf and create a softlink to hive-core.xml file
*cd $SPARK_HOME*
hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6>
*cd conf*hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6/conf> ls -ltr
lrwxrwxrwx 1 hduser hadoop 32 May 3 17:48 hive-site.xml ->
/usr/lib/hive/conf/hive-site.xml
-
You can see the softlink in mine. Just create one as below
ln -s /usr/lib/hive/conf/hive-site.xml hive-site.xml
That should work
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 21 May 2016 at 00:57, Kali.tummala@gmail.com <Ka...@gmail.com>
wrote:
> Hi All ,
>
> Is there a way to ask spark and spark-sql to use Hive 0.14 version instead
> of inbuilt hive 1.2.1.
>
> I am testing spark-sql locally by downloading spark 1.6 from internet , I
> want to execute my hive queries in spark sql using hive version 0.14 can I
> go back to previous version just for a simple test.
>
> Please share out the steps involved.
>
>
> Thanks
> Sri
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-1-6-with-Hive-0-14-tp26989.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>