You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daniel Waeber (JIRA)" <ji...@apache.org> on 2017/04/19 08:06:41 UTC

[jira] [Created] (SPARK-20384) value classes on primitives in DataSets

Daniel Waeber created SPARK-20384:
-------------------------------------

             Summary: value classes on primitives in DataSets
                 Key: SPARK-20384
                 URL: https://issues.apache.org/jira/browse/SPARK-20384
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer, SQL
    Affects Versions: 2.1.0
            Reporter: Daniel Waeber
            Priority: Minor


As a spark user who uses value classes in scala for modelling domain objects, I also would like to make use of them for datasets. 

For example, I would like to use the {{User}} case class which is using a value-class for it's {{id}} as the type for a DataSet:
- the underlying primitive should be mapped to the value-class column
- function on the column (for example comparison ) should only work if defined on the value-class and use these implementation
- show() should pick up the toString method of the value-class

{code}
case class Id(value: Long) extends AnyVal {
  def toString: String = value.toHexString
}
case class User(id: Id, name: String)

spark.sparkContext
  .parallelize(0L to 10L).map(i => (i, f"name-$i")).toDS()
  .withColumnRenamed("_1", "id")
  .withColumnRenamed("_2", "name").as[User]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org