You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daniel Waeber (JIRA)" <ji...@apache.org> on 2017/04/19 08:06:41 UTC
[jira] [Created] (SPARK-20384) value classes on primitives in
DataSets
Daniel Waeber created SPARK-20384:
-------------------------------------
Summary: value classes on primitives in DataSets
Key: SPARK-20384
URL: https://issues.apache.org/jira/browse/SPARK-20384
Project: Spark
Issue Type: Improvement
Components: Optimizer, SQL
Affects Versions: 2.1.0
Reporter: Daniel Waeber
Priority: Minor
As a spark user who uses value classes in scala for modelling domain objects, I also would like to make use of them for datasets.
For example, I would like to use the {{User}} case class which is using a value-class for it's {{id}} as the type for a DataSet:
- the underlying primitive should be mapped to the value-class column
- function on the column (for example comparison ) should only work if defined on the value-class and use these implementation
- show() should pick up the toString method of the value-class
{code}
case class Id(value: Long) extends AnyVal {
def toString: String = value.toHexString
}
case class User(id: Id, name: String)
spark.sparkContext
.parallelize(0L to 10L).map(i => (i, f"name-$i")).toDS()
.withColumnRenamed("_1", "id")
.withColumnRenamed("_2", "name").as[User]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org