You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "gileny@gmail.com" <gi...@gmail.com> on 2014/07/09 13:28:00 UTC
Spark SQL - java.lang.NoClassDefFoundError: Could not initialize
class $line10.$read$
Hello,While trying to run this example below I am getting errors.I have build
Spark using the followng command:$ SPARK_HADOOP_VERSION=2.4.0
SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean
assembly-----------------------------------------Running the example using
Spark-shell---------------------------------------$
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.4.0.jar
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client
./bin/spark-shellscala>.... val sqlContext = new
org.apache.spark.sql.SQLContext(sc)import sqlContext._case class
Person(name: String, age: Int)val people =
sc.textFile("hdfs://myd-vm05698.hpswlabs.adapps.hp.com:9000/user/spark/examples/people.txt").map(_.split(",")).map(p
=> Person(p(0), p(1).trim.toInt))people.registerAsTable("people")val
teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <=
19")teenagers.map(t => "Name: " +
t(0)).collect().foreach(println)----------------------------------------------error-----------------------------------------------java.lang.NoClassDefFoundError:
Could not initialize class $line10.$read$ at
$line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
$line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at
scala.collection.Iterator$$anon$1.next(Iterator.scala:853) at
scala.collection.Iterator$$anon$1.head(Iterator.scala:840) at
org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:181)
at
org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:176)
at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:112) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:112)
at org.apache.spark.scheduler.Task.run(Task.scala:51) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-java-lang-NoClassDefFoundError-Could-not-initialize-class-line10-read-tp9170.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
RE: Spark SQL - java.lang.NoClassDefFoundError: Could not
initialize class $line10.$read$
Posted by "Peretz, Gil" <gp...@hp.com>.
Thank You, I managed to activate Spark-Shell and to use it, by re-compiling and packaging Spark.
SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean assembly
Regards.
-----------------------------------
Gil Peretz , +972-54-5597107
From: Michael Armbrust [mailto:michael@databricks.com]
Sent: יום ד 09 יולי 2014 21:11
To: user@spark.apache.org
Subject: Re: Spark SQL - java.lang.NoClassDefFoundError: Could not initialize class $line10.$read$
At first glance that looks like an error with the class shipping in the spark shell. (i.e. the line that you type into the spark shell are compiled into classes and then shipped to the executors where they run). Are you able to run other spark examples with closures in the same shell?
Michael
On Wed, Jul 9, 2014 at 4:28 AM, gileny@gmail.com<ma...@gmail.com> <gi...@gmail.com>> wrote:
Hello, While trying to run this example below I am getting errors. I have build Spark using the followng command: $ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean assembly --------------------------------------- --Running the example using Spark-shell --------------------------------------- $ SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.4.0.jar HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client ./bin/spark-shell scala>.... val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext._ case class Person(name: String, age: Int) val people = sc.textFile("hdfs://myd-vm05698.hpswlabs.adapps.hp.com:9000/user/spark/examples/people.txt<http://myd-vm05698.hpswlabs.adapps.hp.com:9000/user/spark/examples/people.txt>").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt)) people.registerAsTable("people") val teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <= 19") teenagers.map(t => "Name: " + t(0)).collect().foreach(println) ---------------------------------------------- error ----------------------------------------------- java.lang.NoClassDefFoundError: Could not initialize class $line10.$read$ at $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$1.next(Iterator.scala:853) at scala.collection.Iterator$$anon$1.head(Iterator.scala:840) at org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:181) at org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:176) at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:112) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:112) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
________________________________
View this message in context: Spark SQL - java.lang.NoClassDefFoundError: Could not initialize class $line10.$read$<http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-java-lang-NoClassDefFoundError-Could-not-initialize-class-line10-read-tp9170.html>
Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
Re: Spark SQL - java.lang.NoClassDefFoundError: Could not initialize
class $line10.$read$
Posted by Michael Armbrust <mi...@databricks.com>.
At first glance that looks like an error with the class shipping in the
spark shell. (i.e. the line that you type into the spark shell are
compiled into classes and then shipped to the executors where they run).
Are you able to run other spark examples with closures in the same shell?
Michael
On Wed, Jul 9, 2014 at 4:28 AM, gileny@gmail.com <gi...@gmail.com> wrote:
> Hello, While trying to run this example below I am getting errors. I have
> build Spark using the followng command: $ SPARK_HADOOP_VERSION=2.4.0
> SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean assembly
> --------------------------------------- --Running the example using
> Spark-shell --------------------------------------- $
> SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.4.0.jar
> HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client
> ./bin/spark-shell scala>.... val sqlContext = new
> org.apache.spark.sql.SQLContext(sc) import sqlContext._ case class
> Person(name: String, age: Int) val people = sc.textFile("hdfs://
> myd-vm05698.hpswlabs.adapps.hp.com:9000/user/spark/examples/people.txt").map(_.split(",")).map(p
> => Person(p(0), p(1).trim.toInt)) people.registerAsTable("people") val
> teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
> teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
> ---------------------------------------------- error
> -----------------------------------------------
> java.lang.NoClassDefFoundError: Could not initialize class $line10.$read$
> at $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
> $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
> scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at
> scala.collection.Iterator$$anon$1.next(Iterator.scala:853) at
> scala.collection.Iterator$$anon$1.head(Iterator.scala:840) at
> org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:181)
> at
> org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:176)
> at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
> org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:112) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:112) at
> org.apache.spark.scheduler.Task.run(Task.scala:51) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ------------------------------
> View this message in context: Spark SQL - java.lang.NoClassDefFoundError:
> Could not initialize class $line10.$read$
> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-java-lang-NoClassDefFoundError-Could-not-initialize-class-line10-read-tp9170.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>