You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Alessandro De Carli <de...@gmail.com> on 2014/05/11 11:07:23 UTC
java.lang.NoSuchMethodError on Java API
Dear All,
I'm new to the whole Spark framework, but already fell in love with it
:). For a research project at the University of Zurich I'm trying to
implement a Matrix Centroid Decomposition in Spark. I'm using the Java
API.
My problem occurs when I try to call a JavaPairRDD.reduce:
"""
java.lang.NoSuchMethodError:
org.apache.spark.api.java.JavaPairRDD.reduce(Lorg/apache/spark/api/java/function/Function2;)Lscala/Tuple2;
"""
I read in a forum post, that the issue here might be that I'm using
the maven 0.9.1 version
(http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-from-Spark-Java-td4937.html)
I downloaded the GIT version of spark and compiled with sbt, but I
don't really know how I can force my java project to use that one
instead of the maven version.
Does anyone have an advice on how I could achieve this? I'm using
Intellij as IDE and the project is setup with maven.
Best Regards
--
Alessandro De Carli
Sonnmattstr. 121
CH-5242 Birr
Email: decarli.ale@gmail.com
Twitter: @a_d_c_
Tel: +41 76 305 75 00
Web: http://www.papers.ch
Re: java.lang.NoSuchMethodError on Java API
Posted by Madhu <ma...@madhu.com>.
I was able to compile your code in Eclipse.
I ran it using the data in your comments, but I also see the
NoSuchMethodError you mentioned.
It seems to run fine until the call to calculateZVector(...)
It appears that org.apache.commons.math3.util.Pair is not Serializable, so
that's one potential problem.
I created a Serializable version of Pair, but that wasn't enough.
Commenting this code:
zVectorRaw.reduce(new Function2<Tuple2<Integer, Double>, Tuple2<Integer,
Double>, Tuple2<Integer, Double>>() {
@Override
public Tuple2<Integer, Double> call(Tuple2<Integer, Double>
integerDoubleTuple2, Tuple2<Integer, Double> integerDoubleTuple22) throws
Exception {
return null;
}
});
Avoids the NoSuchMethodError, so that might be part of your problem.
Then I get a NotSerializableException, so my guess is there's a reference to
something else that's not serializable in that's referenced in that method.
There's a lot of stuff going on in that method, so it's not easy for me to
follow.
I would break it down to more manageable pieces and build it up one step at
a time.
Sorry I couldn't find the problem.
-----
Madhu
https://www.linkedin.com/in/msiddalingaiah
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-on-Java-API-tp5545p5623.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: java.lang.NoSuchMethodError on Java API
Posted by Alessandro De Carli <de...@gmail.com>.
Sure, I uploaded the code on pastebin: http://pastebin.com/90Hynrjh
On Mon, May 12, 2014 at 12:27 AM, Madhu <ma...@madhu.com> wrote:
> No, you don't need to do anything special to get it to run in Eclipse.
> Just add the assembly jar to the build path, create a main method, add your
> code, and click the green "run" button.
>
> Can you post your code here?
> I can try it in my environment.
>
>
>
> -----
> Madhu
> https://www.linkedin.com/in/msiddalingaiah
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-on-Java-API-tp5545p5567.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
--
Alessandro De Carli
Sonnmattstr. 121
CH-5242 Birr
Email: decarli.ale@gmail.com
Twitter: @a_d_c_
Tel: +41 76 305 75 00
Web: http://www.papers.ch
Re: java.lang.NoSuchMethodError on Java API
Posted by Madhu <ma...@madhu.com>.
No, you don't need to do anything special to get it to run in Eclipse.
Just add the assembly jar to the build path, create a main method, add your
code, and click the green "run" button.
Can you post your code here?
I can try it in my environment.
-----
Madhu
https://www.linkedin.com/in/msiddalingaiah
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-on-Java-API-tp5545p5567.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: java.lang.NoSuchMethodError on Java API
Posted by Alessandro De Carli <de...@gmail.com>.
Madhu,
Thank you! I now switched to eclipse and imported the assembly jar, the IDE
successfully finds the imports. But when I try to run my code I get
"java.lang.NoClassDefFoundError:
org/apache/spark/api/java/function/PairFunction" is there anything special
to consider when I want to run my development code directly from eclipse (I
checked the buildpath settings and made sure to completely export the
assembly jar)?
Best Regards
On May 11, 2014 4:48 PM, "Madhu" <ma...@madhu.com> wrote:
> Alessandro,
>
> I'm using Eclipse, IntelliJ settings will be similar.
> I created a standard project, without maven.
>
> For me, the easiest way was to add this jar to my Eclipse project build
> path:
>
> <my spark
> dir>/assembly/target/scala-2.10/spark-assembly-x.x.x-hadoop1.0.4.jar
>
> It works for either Java or Scala plugin.
> That Jar is quite large (~100MB) and has everything in it.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-on-Java-API-tp5545p5552.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: java.lang.NoSuchMethodError on Java API
Posted by Madhu <ma...@madhu.com>.
Alessandro,
I'm using Eclipse, IntelliJ settings will be similar.
I created a standard project, without maven.
For me, the easiest way was to add this jar to my Eclipse project build
path:
<my spark
dir>/assembly/target/scala-2.10/spark-assembly-x.x.x-hadoop1.0.4.jar
It works for either Java or Scala plugin.
That Jar is quite large (~100MB) and has everything in it.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-on-Java-API-tp5545p5552.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.