[jira] [Resolved] (SPARK-40389) Decimals can't upcast as integral types if the cast can overflow

["Gengliang Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-40389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang resolved SPARK-40389.
------------------------------------
    Fix Version/s: 3.3.1
                   3.4.0
       Resolution: Fixed

Issue resolved by pull request 37832
[https://github.com/apache/spark/pull/37832]

> Decimals can't upcast as integral types if the cast can overflow
> ----------------------------------------------------------------
>
>                 Key: SPARK-40389
>                 URL: https://issues.apache.org/jira/browse/SPARK-40389
>             Project: Spark
>          Issue Type: Task
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Gengliang Wang
>            Assignee: Gengliang Wang
>            Priority: Major
>             Fix For: 3.3.1, 3.4.0
>
>
> In Spark SQL, the method "canUpCast" returns true iff we can safely up-cast the `from` type to `to` type without any truncating or precision lose or possible runtime failures.
> Meanwhile, DecimalType(10, 0) is considered as "canUpCast" to Integer type. This is wrong, since casting 9000000000BD as Integer type will overflow.
> As a result:
>  * The optimizer rule SimplifyCasts replies on the method "canUpCast" and it will mistakenly convert "cast(cast(9000000000BD as int) as long)" as "cast(9000000000BD as long)"
>  * The STRICT store assignment policy relies on this method too. With the policy enabled, inserting 9000000000BD into integer columns will pass compiling time check and insert an unexpected value 410065408.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org