You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/28 15:44:26 UTC
[I] Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode [datafusion-comet]
andygrove opened a new issue, #5069:
URL: https://github.com/apache/datafusion-comet/issues/5069
### What is the problem the feature request solves?
The float/double-to-decimal cast is currently marked `Incompatible` in `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411, 420-422`), so it only runs with `spark.comet.expression.Cast.allowIncompatible=true`. One of the divergences that keeps it incompatible: under ANSI mode, Comet throws for NaN and +/-Infinity inputs where Spark returns NULL.
Spark 4.1 (`Cast.scala`, both interpreted and codegen paths) wraps `Decimal(fractional.toDouble(b))` in a catch of `NumberFormatException`, so `BigDecimal.valueOf(NaN)` results in NULL even under ANSI. Only finite values that overflow the target precision throw.
Comet (`native/spark-expr/src/conversion_funcs/numeric.rs:907-937`) nulls NaN/Inf via `unary_opt`, but then, under ANSI, rescans for newly introduced nulls and raises `NumericValueOutOfRange { value: "NaN" | "inf" }`. The native unit test `test_cast_float_to_decimal_ansi_nan_errors` (`numeric.rs:1424-1450`) locks in this incorrect behavior.
### Describe the potential solution
Exempt NaN and +/-Infinity from the ANSI overflow rescan in `cast_floating_point_to_decimal128` (only finite overflowing values should error), and invert the native test. This removes one of the blockers to promoting the cast pair out of `Incompatible`.
### Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Filed as an enhancement rather than a bug because the cast pair is already gated behind `allowIncompatible`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #5069:
URL: https://github.com/apache/datafusion-comet/issues/5069#issuecomment-5861041944
This was fixed by #5136: cast_floating_point_to_decimal128 now returns NULL for NaN and infinity in every eval mode and raises NUMERIC_VALUE_OUT_OF_RANGE only for finite values that overflow, and the native test was inverted to test_cast_float_to_decimal_ansi_nan_and_infinity_are_null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on issue #5069:
URL: https://github.com/apache/datafusion-comet/issues/5069#issuecomment-5637253553
#5136 (671a8dd9e) made this change: `cast_floating_point_to_decimal128` in native/spark-expr/src/conversion_funcs/numeric.rs now treats NaN and +/-Infinity as NULL in every eval mode, including ANSI, and only finite values that overflow the target precision raise `NUMERIC_VALUE_OUT_OF_RANGE`. The test this issue pointed at was inverted and renamed to `test_cast_float_to_decimal_ansi_nan_and_infinity_are_null`. Suggest closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #5069: Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode
URL: https://github.com/apache/datafusion-comet/issues/5069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode [datafusion-comet]
Posted by "Smallfu666 (via GitHub)" <gi...@apache.org>.
Smallfu666 commented on issue #5069:
URL: https://github.com/apache/datafusion-comet/issues/5069#issuecomment-5140889259
It looks like #5136 already implements this: NaN/Infinity now produce null in all eval modes including ANSI (the NumberFormatException from `Decimal(double)` is caught before the ANSI overflow check, matching Spark), and it renames the test this issue points at. So this can probably be closed as resolved by #5136 once that merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org