You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Khalid Reid (JIRA)" <ji...@apache.org> on 2016/10/12 17:05:20 UTC
[jira] [Updated] (SPARK-17890) scala.ScalaReflectionException
[ https://issues.apache.org/jira/browse/SPARK-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Khalid Reid updated SPARK-17890:
--------------------------------
Description:
Hello,
I am seeing a very cryptic error message in spark-shell when I map a DataFrame to a Seq\[Foo\]. However, things work fine when I use flatMap.
{noformat}
scala> case class Foo(value:String)
defined class Foo
scala> val df = sc.parallelize(List(1,2,3)).toDF
df: org.apache.spark.sql.DataFrame = [value: int]
scala> df.map{x => Seq.empty[Foo]}
scala.ScalaReflectionException: object $line14.$read not found.
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
at $typecreator1$1.apply(<console>:29)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125)
... 48 elided
scala> df.flatMap{_ => Seq.empty[Foo]} //flatMap works
res2: org.apache.spark.sql.Dataset[Foo] = [value: string]
{noformat}
I am seeing the same error reported [here|https://issues.apache.org/jira/browse/SPARK-8465?jql=text%20~%20%22scala.ScalaReflectionException%22] when I use spark-submit.
I am new to Spark but I don't expect this to throw an exception.
Thanks.
was:
Hello,
I am seeing a very cryptic error message in spark-shell when I map a DataFrame to a Seq\[Foo\]. However, things work fine when I use flatMap.
{noformat}
scala> case class Foo(value:String)
defined class Foo
scala> val df = sc.parallelize(List(1,2,3)).toDF
df: org.apache.spark.sql.DataFrame = [value: int]
scala> df.map{x => Seq.empty[Foo]}
scala.ScalaReflectionException: object $line14.$read not found.
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
at $typecreator1$1.apply(<console>:29)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125)
... 48 elided
scala> df.flatMap{_ => Seq.empty[Foo]} //flatMap works
res2: org.apache.spark.sql.Dataset[Foo] = [value: string]
{noformat}
I am seeing the same error reported [here|https://issues.apache.org/jira/browse/SPARK-8465?jql=text%20~%20%22scala.ScalaReflectionException%22] when I use spark-submit.
I am new to Spark so if this an invalid code (though I don't see why it would be) then I'd expect to see a better error message.
> scala.ScalaReflectionException
> ------------------------------
>
> Key: SPARK-17890
> URL: https://issues.apache.org/jira/browse/SPARK-17890
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.1
> Environment: x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Reporter: Khalid Reid
> Priority: Minor
> Labels: newbie
>
> Hello,
> I am seeing a very cryptic error message in spark-shell when I map a DataFrame to a Seq\[Foo\]. However, things work fine when I use flatMap.
> {noformat}
> scala> case class Foo(value:String)
> defined class Foo
> scala> val df = sc.parallelize(List(1,2,3)).toDF
> df: org.apache.spark.sql.DataFrame = [value: int]
> scala> df.map{x => Seq.empty[Foo]}
> scala.ScalaReflectionException: object $line14.$read not found.
> at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)
> at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
> at $typecreator1$1.apply(<console>:29)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
> at org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
> at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
> at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125)
> ... 48 elided
> scala> df.flatMap{_ => Seq.empty[Foo]} //flatMap works
> res2: org.apache.spark.sql.Dataset[Foo] = [value: string]
> {noformat}
> I am seeing the same error reported [here|https://issues.apache.org/jira/browse/SPARK-8465?jql=text%20~%20%22scala.ScalaReflectionException%22] when I use spark-submit.
> I am new to Spark but I don't expect this to throw an exception.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org