You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Aravindakshan Srinivasan <s_...@yahoo.com.INVALID> on 2016/07/12 20:11:43 UTC
Spark Action and HiveContext
Team,
Has anybody been successful using HiveContext via the Oozie Spark action? I get errors regardless of how I package and share the spark-hive jars. This is on Oozie 4.2.0.
User class threw exception: java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext Wondering if folks have figured out some patterns or solutions for this.
Best,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan
http://goodcynicsports.blogspot.com
I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr. --------------------------------------------------------------------------------------------
Re: Spark Action and HiveContext
Posted by Aravindakshan Srinivasan <s_...@yahoo.com.INVALID>.
Yeah, we don't build with -Phive on purpose. It should still work though as long as the jars are available and packaged, no?
Aravind
From: Jeff Zhang <zj...@gmail.com>
To: user@oozie.apache.org; Aravindakshan Srinivasan <s_...@yahoo.com>
Sent: Tuesday, July 12, 2016 4:51 PM
Subject: Re: Spark Action and HiveContext
I suspect you didn't enable -Phive when building spark.
On Wed, Jul 13, 2016 at 7:19 AM, Aravindakshan Srinivasan <
s_aravind@yahoo.com.invalid> wrote:
> Thanks Peter, but that doesn't help. The HiveContext class is part of
> Spark-hive jars and not really part of the Hive share-lib. But Oozie just
> doesn't seem to see the spark-hive jar regardless of where I put it.
>
> Aravind
>
> From: Peter Cseh <ge...@cloudera.com>
> To: user@oozie.apache.org; Aravindakshan Srinivasan <s_...@yahoo.com>
> Sent: Tuesday, July 12, 2016 1:45 PM
> Subject: Re: Spark Action and HiveContext
>
> Hi,
>
> Have you tried adding the hive sharelib to the action?
> <configuration>
> <property>
> <name>oozie.action.sharelib.for.spark</name>
> <value>spark,hive</value>
> </property>
> </configuration>
> You may define this on action, workflow or global level.
> Peter
>
> On Tue, Jul 12, 2016 at 10:11 PM, Aravindakshan Srinivasan <
> s_aravind@yahoo.com.invalid> wrote:
>
> > Team,
> > Has anybody been successful using HiveContext via the Oozie Spark action?
> > I get errors regardless of how I package and share the spark-hive jars.
> > This is on Oozie 4.2.0.
> > User class threw exception: java.lang.NoClassDefFoundError:
> > org/apache/spark/sql/hive/HiveContext Wondering if folks have figured out
> > some patterns or solutions for this.
> > Best,Aravind
> --------------------------------------------------------------------------------------------Aravind
> > Srinivasan
> > http://goodcynicsports.blogspot.com
> > I have decided to stick with love. Hate is too great a burden to bear.
> > - MLK,
> > Jr.
> --------------------------------------------------------------------------------------------
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>
>
>
>
--
Best Regards
Jeff Zhang
Re: Spark Action and HiveContext
Posted by Jeff Zhang <zj...@gmail.com>.
I suspect you didn't enable -Phive when building spark.
On Wed, Jul 13, 2016 at 7:19 AM, Aravindakshan Srinivasan <
s_aravind@yahoo.com.invalid> wrote:
> Thanks Peter, but that doesn't help. The HiveContext class is part of
> Spark-hive jars and not really part of the Hive share-lib. But Oozie just
> doesn't seem to see the spark-hive jar regardless of where I put it.
>
> Aravind
>
> From: Peter Cseh <ge...@cloudera.com>
> To: user@oozie.apache.org; Aravindakshan Srinivasan <s_...@yahoo.com>
> Sent: Tuesday, July 12, 2016 1:45 PM
> Subject: Re: Spark Action and HiveContext
>
> Hi,
>
> Have you tried adding the hive sharelib to the action?
> <configuration>
> <property>
> <name>oozie.action.sharelib.for.spark</name>
> <value>spark,hive</value>
> </property>
> </configuration>
> You may define this on action, workflow or global level.
> Peter
>
> On Tue, Jul 12, 2016 at 10:11 PM, Aravindakshan Srinivasan <
> s_aravind@yahoo.com.invalid> wrote:
>
> > Team,
> > Has anybody been successful using HiveContext via the Oozie Spark action?
> > I get errors regardless of how I package and share the spark-hive jars.
> > This is on Oozie 4.2.0.
> > User class threw exception: java.lang.NoClassDefFoundError:
> > org/apache/spark/sql/hive/HiveContext Wondering if folks have figured out
> > some patterns or solutions for this.
> > Best,Aravind
> --------------------------------------------------------------------------------------------Aravind
> > Srinivasan
> > http://goodcynicsports.blogspot.com
> > I have decided to stick with love. Hate is too great a burden to bear.
> > - MLK,
> > Jr.
> --------------------------------------------------------------------------------------------
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>
>
>
>
--
Best Regards
Jeff Zhang
Re: Spark Action and HiveContext
Posted by Aravindakshan Srinivasan <s_...@yahoo.com.INVALID>.
Thanks Peter, but that doesn't help. The HiveContext class is part of Spark-hive jars and not really part of the Hive share-lib. But Oozie just doesn't seem to see the spark-hive jar regardless of where I put it.
Aravind
From: Peter Cseh <ge...@cloudera.com>
To: user@oozie.apache.org; Aravindakshan Srinivasan <s_...@yahoo.com>
Sent: Tuesday, July 12, 2016 1:45 PM
Subject: Re: Spark Action and HiveContext
Hi,
Have you tried adding the hive sharelib to the action?
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark,hive</value>
</property>
</configuration>
You may define this on action, workflow or global level.
Peter
On Tue, Jul 12, 2016 at 10:11 PM, Aravindakshan Srinivasan <
s_aravind@yahoo.com.invalid> wrote:
> Team,
> Has anybody been successful using HiveContext via the Oozie Spark action?
> I get errors regardless of how I package and share the spark-hive jars.
> This is on Oozie 4.2.0.
> User class threw exception: java.lang.NoClassDefFoundError:
> org/apache/spark/sql/hive/HiveContext Wondering if folks have figured out
> some patterns or solutions for this.
> Best,Aravind --------------------------------------------------------------------------------------------Aravind
> Srinivasan
> http://goodcynicsports.blogspot.com
> I have decided to stick with love. Hate is too great a burden to bear.
> - MLK,
> Jr. --------------------------------------------------------------------------------------------
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>
Re: Spark Action and HiveContext
Posted by Peter Cseh <ge...@cloudera.com>.
Hi,
Have you tried adding the hive sharelib to the action?
<configuration>
<property>
<name>oozie.action.sharelib.for.spark</name>
<value>spark,hive</value>
</property>
</configuration>
You may define this on action, workflow or global level.
Peter
On Tue, Jul 12, 2016 at 10:11 PM, Aravindakshan Srinivasan <
s_aravind@yahoo.com.invalid> wrote:
> Team,
> Has anybody been successful using HiveContext via the Oozie Spark action?
> I get errors regardless of how I package and share the spark-hive jars.
> This is on Oozie 4.2.0.
> User class threw exception: java.lang.NoClassDefFoundError:
> org/apache/spark/sql/hive/HiveContext Wondering if folks have figured out
> some patterns or solutions for this.
> Best,Aravind --------------------------------------------------------------------------------------------Aravind
> Srinivasan
> http://goodcynicsports.blogspot.com
> I have decided to stick with love. Hate is too great a burden to bear.
> - MLK,
> Jr. --------------------------------------------------------------------------------------------
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>