[jira] [Created] (SPARK-22663) Spark DataSet to case class mapping mismatches

["Sajeev Ramakrishnan (JIRA)" <ji...@apache.org>]

Sajeev Ramakrishnan created SPARK-22663:
-------------------------------------------

             Summary: Spark DataSet to case class mapping mismatches
                 Key: SPARK-22663
                 URL: https://issues.apache.org/jira/browse/SPARK-22663
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.2.0
            Reporter: Sajeev Ramakrishnan
            Priority: Minor


Dear Team,
  As of now when we create a Dataset from a datasource, we give as[<case-class>] at the end to do the mapping. But if the case class is having an extra attribute, then spark throws error.

Eg. 

case class MyClass(
                var line: String = "",
                var prevLine: String = ""
)

val raw= spark.read.textFile(<file>)
var a:Dataset[MyClass] = raw.withColumn("line", split(col("value"), "\\t")).select(
      col("line").getItem(0).as("line")
).as[MyClass]

This code fails telling that there is no match for prevLine

Fixing this would be easy to build spark programs with Datasets where so many joins are involved and the result would add multiple columns everytime. It will be difficult to have different case classes for different joins.

Thanks & Regards,
Sajeev Ramakrishnan



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org