You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/01/31 10:20:34 UTC
[jira] [Updated] (SPARK-5506) java.lang.ClassCastException using
lambda expressions in combination of spark and Servlet
[ https://issues.apache.org/jira/browse/SPARK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-5506:
-----------------------------
Priority: Major (was: Blocker)
(In case anyone else gets confused for a second, there is another Java project called "Spark", and it is also being used here!)
Interesting, all I can glean from this is that something did not deserialize the lambda expression passed to filter(). Given that it only occurs inside of this web framework, is it not possible it is somehow failing to handle the lambda? Because it indeed works correctly in your example and in unit tests.
> java.lang.ClassCastException using lambda expressions in combination of spark and Servlet
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-5506
> URL: https://issues.apache.org/jira/browse/SPARK-5506
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 1.2.0
> Environment: spark server: Ubuntu 14.04 amd64
> $ java -version
> java version "1.8.0_25"
> Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
> Reporter: Milad Khajavi
>
> I'm trying to build a web API for my Apache spark jobs using sparkjava.com framework. My code is:
> @Override
> public void init() {
> get("/hello",
> (req, res) -> {
> String sourcePath = "hdfs://spark:54310/input/*";
> SparkConf conf = new SparkConf().setAppName("LineCount");
> conf.setJars(new String[] { "/home/sam/resin-4.0.42/webapps/test.war" });
> File configFile = new File("config.properties");
> String sparkURI = "spark://hamrah:7077";
> conf.setMaster(sparkURI);
> conf.set("spark.driver.allowMultipleContexts", "true");
> JavaSparkContext sc = new JavaSparkContext(conf);
> @SuppressWarnings("resource")
> JavaRDD<String> log = sc.textFile(sourcePath);
> JavaRDD<String> lines = log.filter(x -> {
> return true;
> });
> return lines.count();
> });
> }
> If I remove the lambda expression or put it inside a simple jar rather than a web service (somehow a Servlet) it will run without any error. But using a lambda expression inside a Servlet will result this exception:
> 15/01/28 10:36:33 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, hamrah): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDD$$anonfun$filter$1.f$1 of type org.apache.spark.api.java.function.Function in instance of org.apache.spark.api.java.JavaRDD$$anonfun$filter$1
> at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)
> at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1999)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> P.S: I tried combination of jersey and javaspark with jetty, tomcat and resin and all of them led me to the same result.
> Here the same issue: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-YARN-java-lang-ClassCastException-SerializedLambda-to-org-apache-spark-api-java-function-Fu1-tt21261.html
> This is my colleague question in stackoverflow: http://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org