You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by "Huang, Jie" <ji...@intel.com> on 2013/08/14 08:30:12 UTC
Fail to use reduceByKey in latest github version with hadoop2
Hi all,
I am a newbie to Spark. After getting the latest code(0.8.0-SNAPSHOT, the default master branch) from github, and compiling it with Hadoop2, I found it failed to run the following example command demonstrated in the Spark document.
$> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
BTW, I have prepared the package with SBT, and configured the Hadoop version in SparkBuild.scala file. And it doesn’t have the similar problem while choosing hadoop1 option.
It seems the problem happening with the reduceByKey function(or its class loader). Since I have tried those RDD transformations one by one. And you can find more details as below.
Is there any solution to this problem? Is there anything missing in my preparation work? Thank you very much for your time and help.
---------------------------------------------------------------------------------------------------------------------------
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88)
at scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88)
at scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at spark.PairRDDFunctions.<init>(PairRDDFunctions.scala:53)
at spark.SparkContext$.rddToPairRDDFunctions(SparkContext.scala:877)
at <init>(<console>:14)
at <init>(<console>:19)
at <init>(<console>:21)
at <init>(<console>:23)
at <init>(<console>:25)
at .<init>(<console>:29)
at .<clinit>(<console>)
at .<init>(<console>:11)
at .<clinit>(<console>)
at $export(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
at java.lang.Thread.run(Thread.java:662)
Thank you && Best Regards,
Grace （Huang Jie)
Re: Fail to use reduceByKey in latest github version with hadoop2
Posted by Matei Zaharia <ma...@gmail.com>.
Maybe there are some old versions of libraries on the classpath. Run sbt clean and then sbt package again.
Matei
On Aug 13, 2013, at 11:30 PM, "Huang, Jie" <ji...@intel.com> wrote:
> Hi all,
>
> I am a newbie to Spark. After getting the latest code(0.8.0-SNAPSHOT, the default master branch) from github, and compiling it with Hadoop2, I found it failed to run the following example command demonstrated in the Spark document.
> $> val wordCounts = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b)
>
> BTW, I have prepared the package with SBT, and configured the Hadoop version in SparkBuild.scala file. And it doesn’t have the similar problem while choosing hadoop1 option.
>
> It seems the problem happening with the reduceByKey function(or its class loader). Since I have tried those RDD transformations one by one. And you can find more details as below.
> Is there any solution to this problem? Is there anything missing in my preparation work? Thank you very much for your time and help.
>
> ---------------------------------------------------------------------------------------------------------------------------
> java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$findClass(ScalaClassLoader.scala:88)
> at scala.tools.nsc.util.ScalaClassLoader$class.findClass(ScalaClassLoader.scala:44)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.findClass(ScalaClassLoader.scala:88)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.scala$tools$nsc$util$ScalaClassLoader$$super$loadClass(ScalaClassLoader.scala:88)
> at scala.tools.nsc.util.ScalaClassLoader$class.loadClass(ScalaClassLoader.scala:50)
> at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.loadClass(ScalaClassLoader.scala:88)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at spark.PairRDDFunctions.<init>(PairRDDFunctions.scala:53)
> at spark.SparkContext$.rddToPairRDDFunctions(SparkContext.scala:877)
> at <init>(<console>:14)
> at <init>(<console>:19)
> at <init>(<console>:21)
> at <init>(<console>:23)
> at <init>(<console>:25)
> at .<init>(<console>:29)
> at .<clinit>(<console>)
> at .<init>(<console>:11)
> at .<clinit>(<console>)
> at $export(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
> at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
> at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
> at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
> at java.lang.Thread.run(Thread.java:662)
>
> Thank you && Best Regards,
> Grace （Huang Jie)
>
>