You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adrian Mocanu <am...@verticalscope.com> on 2015/01/22 16:57:26 UTC

Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

Hi
I get this exception when I run a Spark test case on my local machine:

An exception or error caused a run to abort: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lorg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/dstream/PairDStreamFunctions;
java.lang.NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lorg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/dstream/PairDStreamFunctions;

In my test case I have these Spark related imports imports:
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.TestSuiteBase
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions

-Adrian


Re: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

Posted by Yana Kadiyska <ya...@gmail.com>.
if you're running the test via sbt you can examine the classpath that sbt
uses for the test (show runtime:full-classpath or last run)-- I find this
helps once too many includes and excludes interact.

On Thu, Jan 22, 2015 at 3:50 PM, Adrian Mocanu <am...@verticalscope.com>
wrote:

>
> I use spark 1.1.0-SNAPSHOT and the test I'm running is in local mode. My
> test case uses org.apache.spark.streaming.TestSuiteBase
>
> val spark="org.apache.spark" %% "spark-core" % "1.1.0-SNAPSHOT" %
> "provided" excludeAll(
> val sparkStreaming= "org.apache.spark" % "spark-streaming_2.10" %
> "1.1.0-SNAPSHOT" % "provided" excludeAll(
> val sparkCassandra= "com.tuplejump" % "calliope_2.10" % "0.9.0-C2-EA"
> exclude("org.apache.cassandra", "cassandra-all")
> exclude("org.apache.cassandra", "cassandra-thrift")
> val casAll = "org.apache.cassandra" % "cassandra-all" % "2.0.3"
> intransitive()
> val casThrift = "org.apache.cassandra" % "cassandra-thrift" % "2.0.3"
> intransitive()
> val sparkStreamingFromKafka = "org.apache.spark" %
> "spark-streaming-kafka_2.10" % "0.9.1" excludeAll(
>
>
> -----Original Message-----
> From: Sean Owen [mailto:sowen@cloudera.com]
> Sent: January-22-15 11:39 AM
> To: Adrian Mocanu
> Cc: user@spark.incubator.apache.org
> Subject: Re: Exception: NoSuchMethodError:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions
>
> NoSuchMethodError almost always means that you have compiled some code
> against one version of a library but are running against another. I wonder
> if you are including different versions of Spark in your project, or
> running against a cluster on an older version?
>
> On Thu, Jan 22, 2015 at 3:57 PM, Adrian Mocanu <am...@verticalscope.com>
> wrote:
> > Hi
> >
> > I get this exception when I run a Spark test case on my local machine:
> >
> >
> >
> > An exception or error caused a run to abort:
> > org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> > rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> > la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> > dstream/PairDStreamFunctions;
> >
> > java.lang.NoSuchMethodError:
> > org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> > rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> > la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> > dstream/PairDStreamFunctions;
> >
> >
> >
> > In my test case I have these Spark related imports imports:
> >
> > import org.apache.spark.streaming.StreamingContext._
> >
> > import org.apache.spark.streaming.TestSuiteBase
> >
> > import org.apache.spark.streaming.dstream.DStream
> >
> > import
> > org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
> >
> >
> >
> > -Adrian
> >
> >
>  B KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB
>   [  X  ܚX K  K[XZ[
>   \ \ ][  X  ܚX P   \ ˘\ X  K ܙ B  ܈ Y  ] [ۘ[    [X[     K[XZ[
>   \ \ Z [     \ ˘\ X  K ܙ B B
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

RE: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

Posted by Adrian Mocanu <am...@verticalscope.com>.
I use spark 1.1.0-SNAPSHOT and the test I'm running is in local mode. My test case uses org.apache.spark.streaming.TestSuiteBase

val spark="org.apache.spark" %% "spark-core" % "1.1.0-SNAPSHOT" % "provided" excludeAll( 
val sparkStreaming= "org.apache.spark" % "spark-streaming_2.10" % "1.1.0-SNAPSHOT" % "provided" excludeAll(
val sparkCassandra= "com.tuplejump" % "calliope_2.10" % "0.9.0-C2-EA" exclude("org.apache.cassandra", "cassandra-all") exclude("org.apache.cassandra", "cassandra-thrift")
val casAll = "org.apache.cassandra" % "cassandra-all" % "2.0.3" intransitive()
val casThrift = "org.apache.cassandra" % "cassandra-thrift" % "2.0.3" intransitive()
val sparkStreamingFromKafka = "org.apache.spark" % "spark-streaming-kafka_2.10" % "0.9.1" excludeAll(


-----Original Message-----
From: Sean Owen [mailto:sowen@cloudera.com]
Sent: January-22-15 11:39 AM
To: Adrian Mocanu
Cc: user@spark.incubator.apache.org
Subject: Re: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

NoSuchMethodError almost always means that you have compiled some code against one version of a library but are running against another. I wonder if you are including different versions of Spark in your project, or running against a cluster on an older version?

On Thu, Jan 22, 2015 at 3:57 PM, Adrian Mocanu <am...@verticalscope.com> wrote:
> Hi
>
> I get this exception when I run a Spark test case on my local machine:
>
>
>
> An exception or error caused a run to abort:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> dstream/PairDStreamFunctions;
>
> java.lang.NoSuchMethodError:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> dstream/PairDStreamFunctions;
>
>
>
> In my test case I have these Spark related imports imports:
>
> import org.apache.spark.streaming.StreamingContext._
>
> import org.apache.spark.streaming.TestSuiteBase
>
> import org.apache.spark.streaming.dstream.DStream
>
> import
> org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
>
>
>
> -Adrian
>
>
B KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB  [  X  ܚX KK[XZ[
 \ \ ][  X  ܚX P \ ˘\X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
 \ \ Z[ \ ˘\X K ܙ B B

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


RE: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

Posted by Adrian Mocanu <am...@verticalscope.com>.
I use spark 1.1.0-SNAPSHOT

val spark="org.apache.spark" %% "spark-core" % "1.1.0-SNAPSHOT" % "provided" excludeAll(
-----Original Message-----
From: Sean Owen [mailto:sowen@cloudera.com] 
Sent: January-22-15 11:39 AM
To: Adrian Mocanu
Cc: user@spark.incubator.apache.org
Subject: Re: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

NoSuchMethodError almost always means that you have compiled some code against one version of a library but are running against another. I wonder if you are including different versions of Spark in your project, or running against a cluster on an older version?

On Thu, Jan 22, 2015 at 3:57 PM, Adrian Mocanu <am...@verticalscope.com> wrote:
> Hi
>
> I get this exception when I run a Spark test case on my local machine:
>
>
>
> An exception or error caused a run to abort:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> dstream/PairDStreamFunctions;
>
> java.lang.NoSuchMethodError:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lo
> rg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lsca
> la/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/
> dstream/PairDStreamFunctions;
>
>
>
> In my test case I have these Spark related imports imports:
>
> import org.apache.spark.streaming.StreamingContext._
>
> import org.apache.spark.streaming.TestSuiteBase
>
> import org.apache.spark.streaming.dstream.DStream
>
> import 
> org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
>
>
>
> -Adrian
>
>

Re: Exception: NoSuchMethodError: org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions

Posted by Sean Owen <so...@cloudera.com>.
NoSuchMethodError almost always means that you have compiled some code
against one version of a library but are running against another. I
wonder if you are including different versions of Spark in your
project, or running against a cluster on an older version?

On Thu, Jan 22, 2015 at 3:57 PM, Adrian Mocanu
<am...@verticalscope.com> wrote:
> Hi
>
> I get this exception when I run a Spark test case on my local machine:
>
>
>
> An exception or error caused a run to abort:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lorg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/dstream/PairDStreamFunctions;
>
> java.lang.NoSuchMethodError:
> org.apache.spark.streaming.StreamingContext$.toPairDStreamFunctions(Lorg/apache/spark/streaming/dstream/DStream;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/streaming/dstream/PairDStreamFunctions;
>
>
>
> In my test case I have these Spark related imports imports:
>
> import org.apache.spark.streaming.StreamingContext._
>
> import org.apache.spark.streaming.TestSuiteBase
>
> import org.apache.spark.streaming.dstream.DStream
>
> import org.apache.spark.streaming.StreamingContext.toPairDStreamFunctions
>
>
>
> -Adrian
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org