You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "vaibhawvipul (via GitHub)" <gi...@apache.org> on 2026/03/23 04:04:53 UTC
[PR] fix: Make SumDecimal/AvgDecimal nullable only when not in ANSI mode or input is nullable [datafusion-comet]
vaibhawvipul opened a new pull request, #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #961 .
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
SumDecimal and AvgDecimal hardcode nullable=true, which is correct in non-ANSI mode where overflows produce null values. However, in ANSI mode overflows throw exceptions instead. If the input is non-nullable, the result should also be non-nullable in ANSI mode since there is no code path that can produce a null.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Added `input_nullable field` to SumDecimal and AvgDecimal structs, determined from the child expression's nullability in the planner.
- `is_nullable()` now returns true only in non-ANSI mode or when the input is nullable. In ANSI mode with non-nullable input, it returns false.
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Added a test in `CometAggregateSuite` that exercises SUM and AVG on decimal columns with both nullable and non-nullable inputs, under both ANSI and non-ANSI modes, using checkSparkAnswerAndOperator to verify results match Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Make SumDecimal/AvgDecimal nullable only when not in ANSI mode or input is nullable [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on code in PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#discussion_r2979284694
##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -1917,6 +1917,43 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("SumDecimal and AvgDecimal nullability depends on ANSI mode and input nullability") {
+ // Non-nullable input: in ANSI mode the result should be non-nullable because overflows
+ // throw exceptions instead of producing nulls.
+ val nonNullableData: Seq[(java.math.BigDecimal, Int)] = Seq(
+ (new java.math.BigDecimal("10.00"), 1),
+ (new java.math.BigDecimal("20.00"), 1),
+ (new java.math.BigDecimal("30.00"), 2))
+
+ // Nullable input: result should always be nullable regardless of ANSI mode.
+ val nullableData: Seq[(java.math.BigDecimal, Int)] = Seq(
+ (new java.math.BigDecimal("10.00"), 1),
+ (null.asInstanceOf[java.math.BigDecimal], 1),
+ (new java.math.BigDecimal("30.00"), 2))
+
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // Test SUM with non-nullable input
+ withParquetTable(nonNullableData, "tbl") {
+ val sumRes = sql("SELECT _2, sum(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(sumRes)
+
+ val avgRes = sql("SELECT _2, avg(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(avgRes)
+ }
+
+ // Test SUM/AVG with nullable input
+ withParquetTable(nullableData, "tbl") {
+ val sumRes = sql("SELECT _2, sum(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(sumRes)
+
+ val avgRes = sql("SELECT _2, avg(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(avgRes)
+ }
Review Comment:
Added this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#issuecomment-4120431498
> rebasing on main should fix the `check-missing-suites` ci failure
@parthchandra thank you for the approval. I think this can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Make SumDecimal/AvgDecimal nullable only when not in ANSI mode or input is nullable [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on code in PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#discussion_r2979285899
##########
native/spark-expr/src/agg_funcs/avg_decimal.rs:
##########
@@ -207,6 +213,17 @@ impl AggregateUDFImpl for AvgDecimal {
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
avg_return_type(self.name(), &arg_types[0])
}
+
+ fn is_nullable(&self) -> bool {
Review Comment:
removed the ANSI behaviour to match spark behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra merged PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#issuecomment-4128005921
Merged. Thanks @vaibhawvipul
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Make SumDecimal/AvgDecimal nullable only when not in ANSI mode or input is nullable [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#discussion_r2978374084
##########
spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala:
##########
@@ -1917,6 +1917,43 @@ class CometAggregateSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("SumDecimal and AvgDecimal nullability depends on ANSI mode and input nullability") {
+ // Non-nullable input: in ANSI mode the result should be non-nullable because overflows
+ // throw exceptions instead of producing nulls.
+ val nonNullableData: Seq[(java.math.BigDecimal, Int)] = Seq(
+ (new java.math.BigDecimal("10.00"), 1),
+ (new java.math.BigDecimal("20.00"), 1),
+ (new java.math.BigDecimal("30.00"), 2))
+
+ // Nullable input: result should always be nullable regardless of ANSI mode.
+ val nullableData: Seq[(java.math.BigDecimal, Int)] = Seq(
+ (new java.math.BigDecimal("10.00"), 1),
+ (null.asInstanceOf[java.math.BigDecimal], 1),
+ (new java.math.BigDecimal("30.00"), 2))
+
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // Test SUM with non-nullable input
+ withParquetTable(nonNullableData, "tbl") {
+ val sumRes = sql("SELECT _2, sum(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(sumRes)
+
+ val avgRes = sql("SELECT _2, avg(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(avgRes)
+ }
+
+ // Test SUM/AVG with nullable input
+ withParquetTable(nullableData, "tbl") {
+ val sumRes = sql("SELECT _2, sum(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(sumRes)
+
+ val avgRes = sql("SELECT _2, avg(_1) FROM tbl GROUP BY _2")
+ checkSparkAnswerAndOperator(avgRes)
+ }
Review Comment:
We can add an additional check to ensure that the nullability is compatible with Spark, for example :
`assert(sumRes.schema.fields(1).nullable == true)`
##########
native/spark-expr/src/agg_funcs/avg_decimal.rs:
##########
@@ -207,6 +213,17 @@ impl AggregateUDFImpl for AvgDecimal {
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
avg_return_type(self.name(), &arg_types[0])
}
+
+ fn is_nullable(&self) -> bool {
Review Comment:
In Spark, `Sum.nullable` and Average.nullable` both return `true` irrespective of Ansi mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#issuecomment-4119959021
rebasing on main should fix the `check-missing-suites` ci failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#issuecomment-4115698194
@parthchandra I have added a test in `CometAggregateSuite` to ensure that our behviour matches with spark, under both modes for nullability behaviour.
I have reverted my changes in the rust code to keep comet's behaviour consistent with spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Test: Add test coverage and documentation for SumDecimal/AvgDecimal nullability behavior [datafusion-comet]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on PR #3766:
URL: https://github.com/apache/datafusion-comet/pull/3766#issuecomment-4123118563
@parthchandra thank you for re-basing. i missed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org