[jira] [Updated] (SPARK-59045) Fix SPJ throw ClassCastException with different data type transforms

["XiDuo You (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XiDuo You updated SPARK-59045:
------------------------------
    Summary: Fix SPJ throw ClassCastException with different data type transforms  (was: Fix ClassCastException in storage-partitioned join with compatible transforms)

> Fix SPJ throw ClassCastException with different data type transforms
> --------------------------------------------------------------------
>
>                 Key: SPARK-59045
>                 URL: https://issues.apache.org/jira/browse/SPARK-59045
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: XiDuo You
>            Priority: Major
>
> When a storage-partitioned join uses compatible transforms whose result types differ (e.g.
> `identity(id)` on one side and `bucket(N, id)` on the other), `GroupPartitionsExec.outputPartitioning`
> reports the original partition expressions with the reduced partition keys. The two have different
> data types, so computing the key ordering throws:
> ```
> java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long
> ```
> The fix makes `KeyedShuffleSpec.reducers` return the reduced expression (a `TransformExpression`)
> together with the reducer, and `GroupPartitionsExec.outputPartitioning` report it instead of the
> original expressions, so the reported data types match the reduced keys. For two transforms (e.g.
> two bucket functions) the original expression is kept so it remains reducible for a subsequent join.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org