You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "snmvaughan (via GitHub)" <gi...@apache.org> on 2026/10/23 21:37:19 UTC
[I] Native string to negative-scale decimal cast doesn't match Spark 4.0.5, 4.1.4 and 4.2.1 (SPARK-60119) [datafusion-comet]
snmvaughan opened a new issue, #7187:
URL: https://github.com/apache/datafusion-comet/issues/7187
### Describe the bug
Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), include [SPARK-60119](https://issues.apache.org/jira/browse/SPARK-60119) (apache/spark#59335). When the scale change is over 38, `Decimal.changePrecision` no longer calls `BigDecimal.setScale` if the value's magnitude already decides the result: a value below 0.01 ulp of the target scale is rounded as 0.01 ulp of the same sign.
With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, that changes the negative-scale case of #6966. `'1e-2147483647'` cast to `DECIMAL(10,-2)` returns 0 in every eval mode, where earlier releases throw `ArithmeticException: Underflow`. The native cast still returns NULL, or raises `CAST_INVALID_INPUT` under ANSI, because `parse_string_to_decimal` rejects a scale difference that doesn't fit in an int.
The positive-scale case of #6966 now matches: for `'1e-2147483647'` cast to `DECIMAL(10,2)`, Spark returns `0.00`, as the native cast already does (see #7171).
### Steps to reproduce
On a Spark build with SPARK-60119 (branch-4.0, branch-4.1, branch-4.2 or the release candidates above), with Comet enabled:
```sql
SET spark.sql.legacy.allowNegativeScaleOfDecimal=true;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES ('1e-2147483647');
SELECT try_cast(s AS DECIMAL(10, -2)) FROM t;
-- Spark: 0
-- Comet: NULL
```
With ANSI enabled, `CAST(s AS DECIMAL(10, -2))` returns 0 in Spark and raises `CAST_INVALID_INPUT` in Comet.
### Expected behavior
On releases with SPARK-60119, Comet returns what Spark does: the rounded value, 0 here, for a string far below the target scale's ulp. Earlier releases throw a raw `ArithmeticException`, which the native cast can't reproduce, so it would keep failing under ANSI and returning NULL otherwise there.
### Additional context
SPARK-60119 leaves nothing in the plan for Comet to detect, so the fix would choose the behavior by Spark version, as #7171 does for SPARK-58821 and SPARK-60118 in `SparkFixes`. All three are in the same releases.
`CometNativeCastSuite`'s `(ansi) cast string to negative-scale decimal fails when rescaling overflows an int` expects Spark to fail on this input, so it fails against builds with SPARK-60119.
### Willingness to contribute
I can contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native string to negative-scale decimal cast doesn't match Spark 4.0.5, 4.1.4 and 4.2.1 (SPARK-60119) [datafusion-comet]
Posted by "snmvaughan (via GitHub)" <gi...@apache.org>.
snmvaughan commented on issue #7187:
URL: https://github.com/apache/datafusion-comet/issues/7187#issuecomment-6418407182
From the #7171 review: a second test fails on releases with SPARK-60119. `CometNativeCastSuite`'s `(ansi) failing cast of a literal raises Spark's own exception when reached` expects Spark to fail on `CAST('1e-2147483647' AS DECIMAL(10,2))`, which now returns `0.00`. The fix for this issue should update it too, for example with a literal that fails on every Spark version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native string to negative-scale decimal cast doesn't match Spark 4.0.5, 4.1.4 and 4.2.1 (SPARK-60119) [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #7187:
URL: https://github.com/apache/datafusion-comet/issues/7187#issuecomment-6422734567
We can also fallback first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org