You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Du Li <li...@yahoo-inc.com.INVALID> on 2014/09/11 20:25:59 UTC

SparkSQL HiveContext TypeTag compile error

Hi,

I have the following code snippet. It works fine on spark-shell but in a standalone app it reports "No TypeTag available for MySchema” at compile time when calling hc.createScheamaRdd(rdd). Anybody knows what might be missing?

Thanks,
Du

------
Import org.apache.spark.sql.hive.HiveContext

case class MySchema(key: Int, value: String)
val rdd = sc.parallelize((1 to 10).map(i => MySchema(i, s"val$i")))
val schemaRDD = hc.createSchemaRDD(rdd)
schemaRDD.registerTempTable("data")
val rows = hc.sql("select * from data")
rows.collect.foreach(println)

Re: SparkSQL HiveContext TypeTag compile error

Posted by Du Li <li...@yahoo-inc.com.INVALID>.
Just moving it out of test is not enough. Must move the case class definition to the top level. Otherwise it would report a runtime error of  task not serializable when executing collect().


From: Du Li <li...@yahoo-inc.com.INVALID>>
Date: Thursday, September 11, 2014 at 12:33 PM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: SparkSQL HiveContext TypeTag compile error

Solved it.

The problem occurred because the case class was defined within a test case in FunSuite. Moving the case class definition out of test fixed the problem.


From: Du Li <li...@yahoo-inc.com.INVALID>>
Date: Thursday, September 11, 2014 at 11:25 AM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: SparkSQL HiveContext TypeTag compile error

Hi,

I have the following code snippet. It works fine on spark-shell but in a standalone app it reports "No TypeTag available for MySchema” at compile time when calling hc.createScheamaRdd(rdd). Anybody knows what might be missing?

Thanks,
Du

------
Import org.apache.spark.sql.hive.HiveContext

case class MySchema(key: Int, value: String)
val rdd = sc.parallelize((1 to 10).map(i => MySchema(i, s"val$i")))
val schemaRDD = hc.createSchemaRDD(rdd)
schemaRDD.registerTempTable("data")
val rows = hc.sql("select * from data")
rows.collect.foreach(println)

Re: SparkSQL HiveContext TypeTag compile error

Posted by Du Li <li...@yahoo-inc.com.INVALID>.
Solved it.

The problem occurred because the case class was defined within a test case in FunSuite. Moving the case class definition out of test fixed the problem.


From: Du Li <li...@yahoo-inc.com.INVALID>>
Date: Thursday, September 11, 2014 at 11:25 AM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: SparkSQL HiveContext TypeTag compile error

Hi,

I have the following code snippet. It works fine on spark-shell but in a standalone app it reports "No TypeTag available for MySchema” at compile time when calling hc.createScheamaRdd(rdd). Anybody knows what might be missing?

Thanks,
Du

------
Import org.apache.spark.sql.hive.HiveContext

case class MySchema(key: Int, value: String)
val rdd = sc.parallelize((1 to 10).map(i => MySchema(i, s"val$i")))
val schemaRDD = hc.createSchemaRDD(rdd)
schemaRDD.registerTempTable("data")
val rows = hc.sql("select * from data")
rows.collect.foreach(println)