[jira] [Resolved] (SPARK-49961) Dataset.transform no longer has the correct return type

["Herman van Hövell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-49961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Herman van Hövell resolved SPARK-49961.
---------------------------------------
    Fix Version/s: 4.0.0
       Resolution: Fixed

Issue resolved by pull request 48479
[https://github.com/apache/spark/pull/48479]

> Dataset.transform no longer has the correct return type
> -------------------------------------------------------
>
>                 Key: SPARK-49961
>                 URL: https://issues.apache.org/jira/browse/SPARK-49961
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Chris Twiner
>            Assignee: Chris Twiner
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> In versions prior to 4.0.0-preview2 sql.Dataset transform had signature:
> {code:java}
> def transform[U](t: (sql.Dataset[T]) ⇒ sql.Dataset[U]): sql.Dataset[U] {code}
> 4.0.0-preview2 has moved this to the parent class sql.api.Dataset with the signature:
> {code:java}
> def transform[U](t: (sql.api.Dataset[T]) ⇒ sql.api.Dataset[U]): sql.api.Dataset[U] {code}
> rendering all function objects and return values with incompatible types.
> It seems F Bounded or some similar self type is needed to have the types remain correct (e.g. if you are dealing with sql.Dataset all types should be sql.Dataset),
> {code:java}
> import sparkSession.implicits._
> val ds = Seq(1, 2).toDS()
> val f: Dataset[Int] => Dataset[Int] = d => d.selectExpr("(value + 1) value").as[Int]
> val transformed = ds.transform(f)
> assert(transformed.collect().sorted === Array(2, 3)) {code}
> now fails to compile.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org