You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "0lai0 (via GitHub)" <gi...@apache.org> on 2026/10/07 21:13:52 UTC
[PR] fix: match Spark's ANSI integral SUM overflow error [datafusion-comet]
0lai0 opened a new pull request, #6764:
URL: https://github.com/apache/datafusion-comet/pull/6764
## Which issue does this PR close?
Closes #6661
## Rationale for this change
With `spark.sql.ansi.enabled=true`, Comet raised `ARITHMETIC_OVERFLOW` on an integral `SUM` overflow with the message `integer overflow` and an empty `alternative`. Spark reports `long overflow` with the suggestion `Use 'try_add' to tolerate overflow and return NULL instead.`
Spark's `SUM` over integral input always returns `LONG` and adds through `Add(left, right, evalContext)`. The overflow therefore goes through `MathUtils.addExact(Long, Long, context)`, which passes `try_add` as the hint, whatever the input type. Comet already throws in the same cases. Only the message parameters differ, and that matters to code that inspects `getMessageParameters()` or the message text.
## What changes are included in this PR?
- Add `long_add_overflow_error()` to `native/spark-expr/src/lib.rs`. It returns `SparkError::ArithmeticOverflow { from_type: "long", function_name: "try_add" }`, using the `function_name` field that #6249 added.
- Use it at the three overflow sites in `sum_int.rs`: the ungrouped update (the ungrouped merge reuses it), the grouped update, and the grouped merge. The other callers of `arithmetic_overflow_error` are unchanged.
- `planner.rs` also uses `SumInteger` for window `SUM` over ever-expanding frames, so those windows now report the Spark error too. Sliding frames go through DataFusion's built-in `sum` and are not changed here. That path still ignores ANSI and returns a wrapped value instead of throwing, which #6043 tracks.
## How are these changes tested?
- New Rust unit tests in `sum_int.rs` pin `from_type == "long"` and `function_name == "try_add"` on every overflow path.
- The two ANSI `SUM` tests in `CometAggregateSuite` used to pass with or without the fix. They now use `checkSparkError` and check that `alternative` carries `try_add`. They cover overflow and underflow, grouped and ungrouped queries, and overflow in the partial aggregate and in the final merge.
- Both Scala tests pass under `-Pspark-3.4` and `-Pspark-4.1`, and fail with the native change reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match Spark's ANSI integral SUM overflow error [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on PR #6764:
URL: https://github.com/apache/datafusion-comet/pull/6764#issuecomment-6085356626
Closing this in favor of #6732, which fixes the same issue at the same three sites. I'll be away for the next few days and can't follow up on reviews, so it makes more sense to consolidate there. Thanks @comphead for the review. I've summarized your points on #6732 so they carry over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match Spark's ANSI integral SUM overflow error [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 closed pull request #6764: fix: match Spark's ANSI integral SUM overflow error
URL: https://github.com/apache/datafusion-comet/pull/6764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: match Spark's ANSI integral SUM overflow error [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6764:
URL: https://github.com/apache/datafusion-comet/pull/6764#discussion_r4231743466
##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -3297,36 +3297,49 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
(1 to 50).flatMap(_ => Seq((maxDec38_0, 1)))
}
+ /** Spark's integral `SUM` overflow suggests `try_add`, so Comet's error must too. */
+ private def assertTryAddSuggestion(error: SparkThrowable, clue: String): Unit = {
+ val alternative = error.getMessageParameters.get("alternative")
+ assert(
+ alternative != null && alternative.contains("'try_add'"),
+ s"$clue -> alternative=$alternative")
Review Comment:
#6661 asks for this test to compare the message parameters with Spark's. `assertTryAddSuggestion` only checks that Comet's `alternative` mentions `try_add`, so the Scala test would still pass if `message` came back as `integer overflow`. Only the Rust tests pin `from_type`. `checkSparkError` in `CometTestBase` already holds Spark's error as `expected`, and `CometExpressionSuite` and `CometTemporalExpressionSuite` each hand-roll the `getMessageParameters` comparison today. Could `checkSparkError` take a flag to also assert `actual.getMessageParameters == expected.getMessageParameters`? That checks `message` against Spark on every profile, including 4.2's `overflow`, without hard-coding the wording.
##########
native/spark-expr/src/agg_funcs/sum_int.rs:
##########
@@ -1017,4 +1020,87 @@ mod tests {
acc.merge_batch(&[states]).unwrap();
assert_eq!(acc.evaluate().unwrap(), ScalarValue::Int64(Some(60)));
}
+
+ /// Spark's integral `SUM` adds through `Add` on `LONG`, so every ANSI overflow path must
+ /// report a long overflow with the `try_add` suggestion.
+ fn assert_long_add_overflow(error: DataFusionError) {
+ let DataFusionError::External(error) = error else {
+ panic!("Expected structured Spark error, got {error:?}")
+ };
+ match error.downcast_ref::<SparkError>() {
+ Some(SparkError::ArithmeticOverflow {
+ from_type,
+ function_name,
+ }) => {
+ assert_eq!(from_type, "long");
+ assert_eq!(function_name, "try_add");
+ }
+ other => panic!("Expected ArithmeticOverflow, got {other:?}"),
+ }
+ }
+
+ fn int64_array(values: Vec<i64>) -> ArrayRef {
+ Arc::new(Int64Array::from(values))
+ }
+
+ #[test]
+ fn test_ansi_accumulator_update_batch_overflow() {
+ let mut acc = SumIntegerAccumulatorAnsi::new();
+ let error = acc
+ .update_batch(&[int64_array(vec![i64::MAX, 1])])
+ .unwrap_err();
+ assert_long_add_overflow(error);
+ }
+
+ #[test]
+ fn test_ansi_accumulator_merge_batch_overflow() {
+ let mut acc = SumIntegerAccumulatorAnsi::new();
+ acc.merge_batch(&[int64_array(vec![i64::MAX])]).unwrap();
+ let error = acc.merge_batch(&[int64_array(vec![1])]).unwrap_err();
+ assert_long_add_overflow(error);
+ }
+
+ #[test]
+ fn test_ansi_accumulator_update_batch_underflow() {
Review Comment:
These six tests reach three `map_err` sites. Underflow takes the same `add_checked` branch as overflow, and `SumIntegerAccumulatorAnsi::merge_batch` only calls `update_batch`. So `test_ansi_accumulator_update_batch_underflow`, `test_ansi_accumulator_merge_batch_overflow` and `test_ansi_groups_accumulator_merge_batch_underflow` each repeat a site another test here already pins, and the Scala test drives both signs through all three sites. Could this be one test per site?
##########
native/spark-expr/src/agg_funcs/sum_int.rs:
##########
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use crate::{arithmetic_overflow_error, EvalMode};
+use crate::{long_add_overflow_error, EvalMode};
Review Comment:
#6069 is open and approved, and it adds a fourth ANSI site in this file: `SlidingSumIntegerAccumulator::evaluate` returns `arithmetic_overflow_error("integer")`. Git merges the two PRs without a conflict, but this line drops that import, so from reading the merged tree I expect whichever lands second not to compile. That sliding frame would also keep reporting `integer overflow` with no `try_add`, because Spark recomputes each sliding frame through the same `Add`. Could that site use `long_add_overflow_error()` too, in whichever PR lands second? I have not built the merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org