You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/28 16:06:00 UTC
[PR] fix: honor fail_on_error in native make_decimal [datafusion-comet]
andygrove opened a new pull request, #5080:
URL: https://github.com/apache/datafusion-comet/pull/5080
## Which issue does this PR close?
Closes #5066.
## Rationale for this change
`MakeDecimal` with `nullOnOverflow = false` (ANSI mode) must fail on overflow, matching Spark's `Decimal.set(unscaled, precision, scale)`, which throws `NUMERIC_VALUE_OUT_OF_RANGE`. The Scala serde already serializes `failOnError = !nullOnOverflow`, but the native registration of `make_decimal` discarded the flag, so Comet returned NULL where Spark throws. Since `MakeDecimal` is generated by the `DecimalAggregates` optimizer rule, the divergence surfaces on decimal aggregations rather than user-written SQL.
## What changes are included in this PR?
- `spark_make_decimal` now takes `fail_on_error` and raises `SparkError::NumericValueOutOfRange` (via the existing `decimal_overflow_error` helper, as `CheckOverflow` already does) when the unscaled long does not fit the target precision. Non-ANSI behavior is unchanged: overflow becomes NULL.
- The `make_decimal` registration in `comet_scalar_funcs.rs` threads `fail_on_error` through instead of dropping it.
- Test-only: `createMakeDecimalColumn` in `ShimCometTestBase` (all Spark version variants) gained an overload that accepts `nullOnOverflow`, since the existing helper hardcoded the null-on-overflow behavior.
## How are these changes tested?
- New Rust unit tests in `make_decimal.rs` covering array and scalar overflow with `fail_on_error` set (error) and unset (NULL), plus NULL input under `fail_on_error`.
- New JVM test `make decimal using DataFrame API - overflow throws when nullOnOverflow=false` in `CometExpressionSuite`, asserting Spark and Comet both fail with the `cannot be represented as Decimal(3, 0)` message. The test was verified to fail before the native fix and pass after it.
This issue was found by an audit run with the `audit-comet-expression` project skill (tracked under epic #5078).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: honor fail_on_error in native make_decimal [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5080:
URL: https://github.com/apache/datafusion-comet/pull/5080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: honor fail_on_error in native make_decimal [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5080:
URL: https://github.com/apache/datafusion-comet/pull/5080#discussion_r3675553373
##########
native/spark-expr/src/math_funcs/internal/make_decimal.rs:
##########
@@ -58,14 +60,83 @@ pub fn spark_make_decimal(
}
}
-/// Convert the input long to decimal with the given maximum precision. If overflows, returns null
-/// instead.
+/// Convert the input long to decimal with the given maximum precision. On overflow, errors when
+/// `fail_on_error` is set (Spark's `nullOnOverflow = false`, i.e. ANSI mode) and returns null
+/// otherwise.
#[inline]
-fn long_to_decimal(v: &Option<i64>, precision: u8, scale: i8) -> Option<i128> {
+fn long_to_decimal(
+ v: &Option<i64>,
+ precision: u8,
+ scale: i8,
+ fail_on_error: bool,
+) -> DataFusionResult<Option<i128>> {
match v {
Review Comment:
the block can be rewritten more concise
```
v.map_or(Ok(None), |v| {
let v = v as i128;
match validate_decimal_precision(v, precision, scale) {
Ok(()) => Ok(Some(v)),
Err(_) if fail_on_error => Err(DataFusionError::External(Box::new(
decimal_overflow_error(v, precision, scale),
))),
Err(_) => Ok(None),
}
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org