[jira] [Assigned] (SPARK-60119) Casting a string to decimal fails or is slow when the exponent is far outside the target scale

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun reassigned SPARK-60119:
-------------------------------------

    Assignee: L. C. Hsieh

> Casting a string to decimal fails or is slow when the exponent is far outside the target scale
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60119
>                 URL: https://issues.apache.org/jira/browse/SPARK-60119
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> `Decimal.changePrecision` rescales its BigDecimal value with `setScale` without bounding the scale change. Its cost is proportional to the change, and it throws once the change exceeds the Int range:
> 1. `CAST('1e-2147483647' AS DECIMAL(10,2))` throws `java.lang.ArithmeticException: BigInteger would overflow supported range` under LEGACY, ANSI and TRY, instead of returning 0.00. `'1e-100000000'` returns 0.00 but takes ~40s, since it materializes 10^100000000.
> 2. With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, `fromString` intentionally skips the integral-digit fast-fail, so `CAST('1e2147483647' AS DECIMAL(10,2))` throws `java.lang.ArithmeticException: Underflow` instead of returning NULL (non-ANSI/TRY) or `NUMERIC_VALUE_OUT_OF_RANGE` (ANSI). `'1e100000000'` is also very slow. When `changePrecision` does report the overflow under ANSI, the error message renders the value with `toPlainString`, which cannot fit `1E+2147483647` in a String.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org