You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Stephen Boesch <ja...@gmail.com> on 2014/12/13 03:40:48 UTC
sbt assembly with hive
What is the proper way to build with hive from sbt? The SPARK_HIVE is
deprecated. However after running the following:
sbt -Pyarn -Phadoop-2.3 -Phive assembly/assembly
And then
bin/pyspark
hivectx = HiveContext(sc)
hivectx.hiveql("select * from my_table")
Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and
run sbt/sbt assembly", Py4JError(u'Trying to call a package.',))
Re: sbt assembly with hive
Posted by Abhi Basu <90...@gmail.com>.
I am getting the same message when trying to get HIveContext in CDH 5.1
after enabling Spark. I am thinking Spark should come with Hive enabled
(default option) as Hive metastore is a common way to share data, due to
popularity of Hive and other SQL-Over-Hadoop technologies like Impala.
Thanks,
Abhi
On Fri, Dec 12, 2014 at 6:40 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
>
> What is the proper way to build with hive from sbt? The SPARK_HIVE is
> deprecated. However after running the following:
>
> sbt -Pyarn -Phadoop-2.3 -Phive assembly/assembly
>
> And then
> bin/pyspark
>
> hivectx = HiveContext(sc)
>
> hivectx.hiveql("select * from my_table")
>
> Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and
> run sbt/sbt assembly", Py4JError(u'Trying to call a package.',))
>
--
Abhi Basu