[jira] [Commented] (SPARK-16792) Dataset containing a Case Class with a List type causes a CompileException (converting sequence to list)

["Michal Šenkýř (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735476#comment-15735476 ] 

Michal Šenkýř commented on SPARK-16792:
---------------------------------------

Found a simpler example:
{code:none}
case class X(l: List[String])  // works for Seq
spark.createDataset(Seq(List("A"))).map(X).show
{code}

Exception thrown:
{code:none}
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to scala.collection.immutable.List
  at X$.apply(<console>:11)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
  at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
{code}

> Dataset containing a Case Class with a List type causes a CompileException (converting sequence to list)
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16792
>                 URL: https://issues.apache.org/jira/browse/SPARK-16792
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Jamie Hutton
>            Priority: Critical
>
> The issue occurs when we run a .map over a dataset containing Case Class with a List in it. A self contained test case is below:
> case class TestCC(key: Int, letters: List[String]) //List causes the issue - a Seq/Array works fine
> /*simple test data*/
> val ds1 = sc.makeRDD(Seq(
> (List("D")),
> (List("S","H")),
> (List("F","H")),
> (List("D","L","L"))
> )).map(x=>(x.length,x)).toDF("key","letters").as[TestCC]
> //This will fail
> val test1=ds1.map{_.key}
> test1.show
> Error: 
> Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 72, Column 70: No applicable constructor/method found for actual parameters "int, scala.collection.Seq"; candidates are: "TestCC(int, scala.collection.immutable.List)"
> It seems to be internally converting the List to a sequence, then it cant convert it back...
> If you change the List[String] to Seq[String] or Array[String] the issue doesnt appear



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org