You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/04 23:34:00 UTC
[jira] [Updated] (SPARK-58316) ANSI mode does not throw when casting NaN/Infinity float or double to DECIMAL
[ https://issues.apache.org/jira/browse/SPARK-58316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-58316:
-----------------------------------
Labels: pull-request-available (was: )
> ANSI mode does not throw when casting NaN/Infinity float or double to DECIMAL
> -----------------------------------------------------------------------------
>
> Key: SPARK-58316
> URL: https://issues.apache.org/jira/browse/SPARK-58316
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Philo He
> Priority: Major
> Labels: pull-request-available
>
> h3. Summary
> Under ANSI mode, casting a floating-point value (float/double) to DECIMAL returns NULL for {{NaN}}, {{Infinity}}, and {{-Infinity}}, instead of throwing an error. This is inconsistent with how ANSI mode handles other values that cannot be represented in the target decimal, which throw an exception.
> h3. How to reproduce
> {code:sql} SET spark.sql.ansi.enabled=true;
> -- Finite value that overflows the target precision/scale: throws (expected under ANSI) SELECT CAST(CAST(1e38 AS DOUBLE) AS DECIMAL(20, 2)); -- ArithmeticException
> -- Non-finite values: return NULL instead of throwing SELECT CAST(CAST('nan' AS DOUBLE) AS DECIMAL(38, 2)); -- NULL SELECT CAST(CAST('inf' AS DOUBLE) AS DECIMAL(38, 2)); -- NULL SELECT CAST(CAST('-inf' AS DOUBLE) AS DECIMAL(38, 2)); -- NULL {code}
> h3. Expected behavior
> Under ANSI mode, casting {{NaN}}, {{Infinity}}, or {{-Infinity}} to DECIMAL should throw an error (e.g. a cast overflow / invalid value error), consistent with:
> Finite floating-point values that overflow the target decimal precision/scale, which already throw under ANSI.
> Casting {{NaN}}/{{Infinity}} float/double to integral types, which throws under ANSI.
> A DECIMAL can only represent finite numbers, so {{NaN}} and {{Infinity}} have no valid decimal value. ANSI semantics call for raising a data exception when a value cannot be represented in the target type, rather than silently returning NULL.
> When ANSI mode is disabled, returning NULL is correct and should not change.
> h3. Root cause
> In {{Cast.scala}}, {{castToDecimal}} for {{FractionalType}} wraps the conversion in a try/catch that swallows {{NumberFormatException}} and returns NULL unconditionally, regardless of the ANSI setting:
> {code:scala} case x: FractionalType => val fractional = PhysicalFractionalType.fractional(x) b => try { changePrecision(Decimal(fractional.toDouble(b)), target) } catch { case _: NumberFormatException => null } {code}
> {{Decimal(Double.NaN)}} / {{Decimal(Double.PositiveInfinity)}} construct a Java {{BigDecimal}}, which has no representation for NaN/Infinity and throws {{NumberFormatException}}. Because this exception is caught before {{changePrecision}} (the ANSI-aware code that decides throw-vs-null) runs, the ANSI setting never takes effect for non-finite inputs.
> By contrast, the {{IntegralType}} branch has no such catch, so overflow flows through {{changePrecision}} and throws under ANSI as expected.
> h3. Proposed fix
> Make the catch clause ANSI-aware: when {{ansiEnabled}} is true, throw a proper cast error (e.g. {{castingCauseOverflowError}} / an invalid-input cast error) for non-finite inputs; when ANSI is disabled, return NULL as today. The try/catch itself should remain, since a raw {{NumberFormatException}} must not escape.
> If the behavior change is a compatibility concern, it could be gated behind a legacy config so existing queries can opt back into the NULL behavior.
> h3. Notes
> This was found while implementing float/double → DECIMAL casting in a native engine (Velox, used by Gluten). The overflow cases align with Spark, but the NaN/Infinity handling differs; we chose to follow ANSI semantics (throw under ANSI) and are filing this to check whether Spark intends to align.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org