You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by prai100 <pr...@gmail.com> on 2015/04/10 04:47:55 UTC
ClassCastException when calling updateStateKey
Hi,
We are using Java to write a Spark Streaming job. Everything works fine
if I run standalone. As, soon as I do spark-submit (even if masterURL is
local), I get the following exception -
Job aborted due to stage failure: Task 1 in stage 7754.0 failed 1 times,
most recent failure: Lost task 1.0 in stage 7754.0 (TID 571, localhost):
java.lang.ClassCastException: com.google.common.base.Absent cannot be cast
to com.google.common.base.Optional
Since, Absent is a subclass of Optional this should never happen. But, it
looks like some classLoader issue.
I am hoping somebody else has faced the same issue.
I can provide more details.
Thanks,
PR.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassCastException-when-calling-updateStateKey-tp22448.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: ClassCastException when calling updateStateKey
Posted by Ted Yu <yu...@gmail.com>.
Spark uses Guava 14.0.1
Is it possible that your job includes other version of Guava ?
BTW a brief search for com.google.common.base.Absent in Spark didn't turn
up any hit.
Can you share the complete stack trace ?
Cheers
On Thu, Apr 9, 2015 at 7:47 PM, prai100 <pr...@gmail.com> wrote:
> Hi,
> We are using Java to write a Spark Streaming job. Everything works fine
> if I run standalone. As, soon as I do spark-submit (even if masterURL is
> local), I get the following exception -
> Job aborted due to stage failure: Task 1 in stage 7754.0 failed 1 times,
> most recent failure: Lost task 1.0 in stage 7754.0 (TID 571, localhost):
> java.lang.ClassCastException: com.google.common.base.Absent cannot be cast
> to com.google.common.base.Optional
>
>
> Since, Absent is a subclass of Optional this should never happen. But, it
> looks like some classLoader issue.
>
> I am hoping somebody else has faced the same issue.
>
> I can provide more details.
>
> Thanks,
> PR.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/ClassCastException-when-calling-updateStateKey-tp22448.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: ClassCastException when calling updateStateKey
Posted by Pradeep Rai <pr...@gmail.com>.
Hi Marcelo,
I am not including Spark's classes. When I used the userClasspathFirst
flag, I started getting those errors.
Been there, done that. Removing guava classes was one of the first things I
tried.
I saw your replies to a similar problem from Sept.
http://apache-spark-developers-list.1001551.n3.nabble.com/guava-version-conflicts-td8480.html
It looks like my issue is the same cause, but different symptoms.
Thanks,
Pradeep.
On Fri, Apr 10, 2015 at 12:51 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> On Fri, Apr 10, 2015 at 10:11 AM, Pradeep Rai <pr...@gmail.com> wrote:
> > I tried the userClasspathFirst flag. I started getting NoClassDeFound
> > Exception for spark classes like Function2, etc.
>
> Wait. Are you including Spark classes in your app's assembly? Don't do
> that...
>
> As for Guava, yeah, the mess around Optional and friends is
> unfortunate. One way you could try to work around it, if excluding
> Spark classes and the userClassPathFirst option doesn't work, is to
> explicitly remove the Optional (and related) classes from your app's
> fat jar, and cross your fingers.
>
> --
> Marcelo
>
Re: ClassCastException when calling updateStateKey
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Fri, Apr 10, 2015 at 10:11 AM, Pradeep Rai <pr...@gmail.com> wrote:
> I tried the userClasspathFirst flag. I started getting NoClassDeFound
> Exception for spark classes like Function2, etc.
Wait. Are you including Spark classes in your app's assembly? Don't do that...
As for Guava, yeah, the mess around Optional and friends is
unfortunate. One way you could try to work around it, if excluding
Spark classes and the userClassPathFirst option doesn't work, is to
explicitly remove the Optional (and related) classes from your app's
fat jar, and cross your fingers.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: ClassCastException when calling updateStateKey
Posted by Pradeep Rai <pr...@gmail.com>.
I tried the userClasspathFirst flag. I started getting NoClassDeFound
Exception for spark classes like Function2, etc.
We actually tried removing guava jar file from our assembly, and that did
not help.
We are using DataStax 2.6 distribution which uses Spark 1.1.
I have attached the exception log.
Thanks for replying.
On Fri, Apr 10, 2015 at 4:36 AM, Sean Owen <so...@cloudera.com> wrote:
> It is because they are in different classloaders. You are providing Absent
> and Spark has Optional but not Absent (long story). Try the user classpath
> first properties.
> On Apr 10, 2015 3:48 AM, "prai100" <pr...@gmail.com> wrote:
>
>> Hi,
>> We are using Java to write a Spark Streaming job. Everything works fine
>> if I run standalone. As, soon as I do spark-submit (even if masterURL is
>> local), I get the following exception -
>> Job aborted due to stage failure: Task 1 in stage 7754.0 failed 1 times,
>> most recent failure: Lost task 1.0 in stage 7754.0 (TID 571, localhost):
>> java.lang.ClassCastException: com.google.common.base.Absent cannot be cast
>> to com.google.common.base.Optional
>>
>>
>> Since, Absent is a subclass of Optional this should never happen. But, it
>> looks like some classLoader issue.
>>
>> I am hoping somebody else has faced the same issue.
>>
>> I can provide more details.
>>
>> Thanks,
>> PR.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/ClassCastException-when-calling-updateStateKey-tp22448.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: ClassCastException when calling updateStateKey
Posted by Sean Owen <so...@cloudera.com>.
It is because they are in different classloaders. You are providing Absent
and Spark has Optional but not Absent (long story). Try the user classpath
first properties.
On Apr 10, 2015 3:48 AM, "prai100" <pr...@gmail.com> wrote:
> Hi,
> We are using Java to write a Spark Streaming job. Everything works fine
> if I run standalone. As, soon as I do spark-submit (even if masterURL is
> local), I get the following exception -
> Job aborted due to stage failure: Task 1 in stage 7754.0 failed 1 times,
> most recent failure: Lost task 1.0 in stage 7754.0 (TID 571, localhost):
> java.lang.ClassCastException: com.google.common.base.Absent cannot be cast
> to com.google.common.base.Optional
>
>
> Since, Absent is a subclass of Optional this should never happen. But, it
> looks like some classLoader issue.
>
> I am hoping somebody else has faced the same issue.
>
> I can provide more details.
>
> Thanks,
> PR.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/ClassCastException-when-calling-updateStateKey-tp22448.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>