You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2017/04/19 08:20:41 UTC
[jira] [Commented] (SPARK-20384) value classes on primitives in
DataSets
[ https://issues.apache.org/jira/browse/SPARK-20384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974301#comment-15974301 ]
Sean Owen commented on SPARK-20384:
-----------------------------------
Can you make the title more descriptive? an example of what happens now that is a problem would help. This sounds like SPARK-17368 or are you saying that works, you just want additional changes?
> value classes on primitives in DataSets
> ---------------------------------------
>
> Key: SPARK-20384
> URL: https://issues.apache.org/jira/browse/SPARK-20384
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer, SQL
> Affects Versions: 2.1.0
> Reporter: Daniel Davis
> Priority: Minor
>
> As a spark user who uses value classes in scala for modelling domain objects, I also would like to make use of them for datasets.
> For example, I would like to use the {{User}} case class which is using a value-class for it's {{id}} as the type for a DataSet:
> - the underlying primitive should be mapped to the value-class column
> - function on the column (for example comparison ) should only work if defined on the value-class and use these implementation
> - show() should pick up the toString method of the value-class
> {code}
> case class Id(value: Long) extends AnyVal {
> def toString: String = value.toHexString
> }
> case class User(id: Id, name: String)
> spark.sparkContext
> .parallelize(0L to 10L).map(i => (i, f"name-$i")).toDS()
> .withColumnRenamed("_1", "id")
> .withColumnRenamed("_2", "name").as[User]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org