You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/04/25 15:36:46 UTC
Defining case class within main method throws "No TypeTag available
for Accounts"
Hi,
I notice buiding with sbt if I define my case class *outside of main method*
like below it works
case class Accounts( TransactionDate: String, TransactionType: String,
Description: String, Value: Double, Balance: Double, AccountName: String,
AccountNumber : String)
object Import_nw_10124772 {
def main(args: Array[String]) {
val conf = new SparkConf().
setAppName("Import_nw_10124772").
setMaster("local[12]").
set("spark.driver.allowMultipleContexts", "true").
set("spark.hadoop.validateOutputSpecs", "false")
val sc = new SparkContext(conf)
// Create sqlContext based on HiveContext
val sqlContext = new HiveContext(sc)
import sqlContext.implicits._
val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
println ("\nStarted at"); sqlContext.sql("SELECT
FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss')
").collect.foreach(println)
//
// Get a DF first based on Databricks CSV libraries ignore column heading
because of column called "Type"
//
val df =
sqlContext.read.format("com.databricks.spark.csv").option("inferSchema",
"true").option("header",
"true").load("hdfs://rhes564:9000/data/stg/accounts/nw/10124772")
//df.printSchema
//
val a = df.filter(col("Date") > "").map(p =>
Accounts(p(0).toString,p(1).toString,p(2).toString,p(3).toString.toDouble,p(4).toString.toDouble,p(5).toString,p(6).toString))
However, if I put that case class with the main method, it throws "No
TypeTag available for Accounts" error
Apparently when case class is defined inside of the method that it is being
used, it is not fully defined at that point.
Is this a bug within Spark?
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
Re: Defining case class within main method throws "No TypeTag
available for Accounts"
Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
Thanks Michael as I gathered for now it is a feature.
On Monday, 25 April 2016, 18:36, Michael Armbrust <mi...@databricks.com> wrote:
When you define a class inside of a method, it implicitly has a pointer to the outer scope of the method. Spark doesn't have access to this scope, so this makes it hard (impossible?) for us to construct new instances of that class.
So, define your classes that you plan to use with Spark at the top level.
On Mon, Apr 25, 2016 at 9:36 AM, Mich Talebzadeh <mi...@gmail.com> wrote:
Hi,
I notice buiding with sbt if I define my case class outside of main method like below it works
case class Accounts( TransactionDate: String, TransactionType: String, Description: String, Value: Double, Balance: Double, AccountName: String, AccountNumber : String)
object Import_nw_10124772 {
def main(args: Array[String]) {
val conf = new SparkConf().
setAppName("Import_nw_10124772").
setMaster("local[12]").
set("spark.driver.allowMultipleContexts", "true").
set("spark.hadoop.validateOutputSpecs", "false")
val sc = new SparkContext(conf)
// Create sqlContext based on HiveContext
val sqlContext = new HiveContext(sc)
import sqlContext.implicits._
val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
println ("\nStarted at"); sqlContext.sql("SELECT FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss') ").collect.foreach(println)
//
// Get a DF first based on Databricks CSV libraries ignore column heading because of column called "Type"
//
val df = sqlContext.read.format("com.databricks.spark.csv").option("inferSchema", "true").option("header", "true").load("hdfs://rhes564:9000/data/stg/accounts/nw/10124772")
//df.printSchema
//
val a = df.filter(col("Date") > "").map(p => Accounts(p(0).toString,p(1).toString,p(2).toString,p(3).toString.toDouble,p(4).toString.toDouble,p(5).toString,p(6).toString))
However, if I put that case class with the main method, it throws "No TypeTag available for Accounts" error
Apparently when case class is defined inside of the method that it is being used, it is not fully defined at that point.
Is this a bug within Spark?
Thanks
Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com
Re: Defining case class within main method throws "No TypeTag
available for Accounts"
Posted by Mich Talebzadeh <mi...@gmail.com>.
BTW is this documented as it seems to be potential issue.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 April 2016 at 18:47, Mich Talebzadeh <mi...@gmail.com>
wrote:
> cheers
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 25 April 2016 at 18:35, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> When you define a class inside of a method, it implicitly has a pointer
>> to the outer scope of the method. Spark doesn't have access to this scope,
>> so this makes it hard (impossible?) for us to construct new instances of
>> that class.
>>
>> So, define your classes that you plan to use with Spark at the top level.
>>
>> On Mon, Apr 25, 2016 at 9:36 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I notice buiding with sbt if I define my case class *outside of main
>>> method* like below it works
>>>
>>>
>>> case class Accounts( TransactionDate: String, TransactionType: String,
>>> Description: String, Value: Double, Balance: Double, AccountName: String,
>>> AccountNumber : String)
>>>
>>> object Import_nw_10124772 {
>>> def main(args: Array[String]) {
>>> val conf = new SparkConf().
>>> setAppName("Import_nw_10124772").
>>> setMaster("local[12]").
>>> set("spark.driver.allowMultipleContexts", "true").
>>> set("spark.hadoop.validateOutputSpecs", "false")
>>> val sc = new SparkContext(conf)
>>> // Create sqlContext based on HiveContext
>>> val sqlContext = new HiveContext(sc)
>>> import sqlContext.implicits._
>>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>> println ("\nStarted at"); sqlContext.sql("SELECT
>>> FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss')
>>> ").collect.foreach(println)
>>> //
>>> // Get a DF first based on Databricks CSV libraries ignore column
>>> heading because of column called "Type"
>>> //
>>> val df =
>>> sqlContext.read.format("com.databricks.spark.csv").option("inferSchema",
>>> "true").option("header",
>>> "true").load("hdfs://rhes564:9000/data/stg/accounts/nw/10124772")
>>> //df.printSchema
>>> //
>>> val a = df.filter(col("Date") > "").map(p =>
>>> Accounts(p(0).toString,p(1).toString,p(2).toString,p(3).toString.toDouble,p(4).toString.toDouble,p(5).toString,p(6).toString))
>>>
>>>
>>> However, if I put that case class with the main method, it throws "No
>>> TypeTag available for Accounts" error
>>>
>>> Apparently when case class is defined inside of the method that it is
>>> being used, it is not fully defined at that point.
>>>
>>> Is this a bug within Spark?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>
>>
>
Re: Defining case class within main method throws "No TypeTag
available for Accounts"
Posted by Mich Talebzadeh <mi...@gmail.com>.
cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 April 2016 at 18:35, Michael Armbrust <mi...@databricks.com> wrote:
> When you define a class inside of a method, it implicitly has a pointer to
> the outer scope of the method. Spark doesn't have access to this scope, so
> this makes it hard (impossible?) for us to construct new instances of that
> class.
>
> So, define your classes that you plan to use with Spark at the top level.
>
> On Mon, Apr 25, 2016 at 9:36 AM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Hi,
>>
>> I notice buiding with sbt if I define my case class *outside of main
>> method* like below it works
>>
>>
>> case class Accounts( TransactionDate: String, TransactionType: String,
>> Description: String, Value: Double, Balance: Double, AccountName: String,
>> AccountNumber : String)
>>
>> object Import_nw_10124772 {
>> def main(args: Array[String]) {
>> val conf = new SparkConf().
>> setAppName("Import_nw_10124772").
>> setMaster("local[12]").
>> set("spark.driver.allowMultipleContexts", "true").
>> set("spark.hadoop.validateOutputSpecs", "false")
>> val sc = new SparkContext(conf)
>> // Create sqlContext based on HiveContext
>> val sqlContext = new HiveContext(sc)
>> import sqlContext.implicits._
>> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>> println ("\nStarted at"); sqlContext.sql("SELECT
>> FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss')
>> ").collect.foreach(println)
>> //
>> // Get a DF first based on Databricks CSV libraries ignore column
>> heading because of column called "Type"
>> //
>> val df =
>> sqlContext.read.format("com.databricks.spark.csv").option("inferSchema",
>> "true").option("header",
>> "true").load("hdfs://rhes564:9000/data/stg/accounts/nw/10124772")
>> //df.printSchema
>> //
>> val a = df.filter(col("Date") > "").map(p =>
>> Accounts(p(0).toString,p(1).toString,p(2).toString,p(3).toString.toDouble,p(4).toString.toDouble,p(5).toString,p(6).toString))
>>
>>
>> However, if I put that case class with the main method, it throws "No
>> TypeTag available for Accounts" error
>>
>> Apparently when case class is defined inside of the method that it is
>> being used, it is not fully defined at that point.
>>
>> Is this a bug within Spark?
>>
>> Thanks
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
Re: Defining case class within main method throws "No TypeTag
available for Accounts"
Posted by Michael Armbrust <mi...@databricks.com>.
When you define a class inside of a method, it implicitly has a pointer to
the outer scope of the method. Spark doesn't have access to this scope, so
this makes it hard (impossible?) for us to construct new instances of that
class.
So, define your classes that you plan to use with Spark at the top level.
On Mon, Apr 25, 2016 at 9:36 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> I notice buiding with sbt if I define my case class *outside of main
> method* like below it works
>
>
> case class Accounts( TransactionDate: String, TransactionType: String,
> Description: String, Value: Double, Balance: Double, AccountName: String,
> AccountNumber : String)
>
> object Import_nw_10124772 {
> def main(args: Array[String]) {
> val conf = new SparkConf().
> setAppName("Import_nw_10124772").
> setMaster("local[12]").
> set("spark.driver.allowMultipleContexts", "true").
> set("spark.hadoop.validateOutputSpecs", "false")
> val sc = new SparkContext(conf)
> // Create sqlContext based on HiveContext
> val sqlContext = new HiveContext(sc)
> import sqlContext.implicits._
> val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
> println ("\nStarted at"); sqlContext.sql("SELECT
> FROM_unixtime(unix_timestamp(), 'dd/MM/yyyy HH:mm:ss.ss')
> ").collect.foreach(println)
> //
> // Get a DF first based on Databricks CSV libraries ignore column
> heading because of column called "Type"
> //
> val df =
> sqlContext.read.format("com.databricks.spark.csv").option("inferSchema",
> "true").option("header",
> "true").load("hdfs://rhes564:9000/data/stg/accounts/nw/10124772")
> //df.printSchema
> //
> val a = df.filter(col("Date") > "").map(p =>
> Accounts(p(0).toString,p(1).toString,p(2).toString,p(3).toString.toDouble,p(4).toString.toDouble,p(5).toString,p(6).toString))
>
>
> However, if I put that case class with the main method, it throws "No
> TypeTag available for Accounts" error
>
> Apparently when case class is defined inside of the method that it is
> being used, it is not fully defined at that point.
>
> Is this a bug within Spark?
>
> Thanks
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>