You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/22 19:49:27 UTC
[I] Incorrect AVG result from empty native partial buffers [datafusion-comet]
sunchao opened a new issue, #5418:
URL: https://github.com/apache/datafusion-comet/issues/5418
### Describe the bug
An empty native scalar AVG partial currently exports `(sum = NULL, count = 0)`. Spark initializes AVG buffers to `(sum = 0, count = 0)` and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result into `NULL`.
The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as `NULL` and preserve decimal overflow behavior.
### Steps to reproduce
With Comet installed and enabled in a fresh local Spark session (for example, `local[4]`), create four separate Parquet scan partitions and filter out all rows in three of them:
```python
import tempfile
for key, value in {
"spark.sql.adaptive.enabled": "false",
"spark.sql.files.maxPartitionBytes": "1048576",
"spark.sql.parquet.filterPushdown": "false",
"spark.comet.scan.enabled": "false",
"spark.comet.convert.parquet.enabled": "true",
"spark.comet.shuffle.enabled": "false",
"spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
spark.conf.set(key, value)
with tempfile.TemporaryDirectory() as tmp:
path = f"{tmp}/data"
(spark.range(8, numPartitions=4)
.selectExpr("id", "cast(id + 1 as int) quantity")
.write.parquet(path))
spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id = 1").show()
```
The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns `NULL` instead of `2.0`.
### Expected behavior
The result is `2.0`, matching Spark. If the filter removes every row, the final result remains `NULL`.
### Additional context
Reproduced against Apache Comet `main` at `2699f59b71788e17a2714910e166a3f83deed937` using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect AVG result from empty native partial buffers [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5418: Incorrect AVG result from empty native partial buffers
URL: https://github.com/apache/datafusion-comet/issues/5418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect AVG result from empty native partial buffers [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #5418:
URL: https://github.com/apache/datafusion-comet/issues/5418#issuecomment-6192492552
I checked this against main at 66c63bd307 on Spark 4.1.3, and the repro no longer returns NULL. Since #5421, a Spark final aggregate keeps the AVG partial below it in Spark too, so the query returns 2.0. That holds with AQE on and off, with the final-mode switch, and with Comet shuffle disabled. 1.1.0 has this guard and branch-1.0 has it through #6422, but 1.0.0 shipped without it.
The native state hasn't changed. An untouched scalar `AvgAccumulator` still exports `(NULL, 0)`. When I let AVG partials feed a Spark final again, the same query returns NULL in every mixed configuration, and the `CometAggregateSuite` test `COUNT and AVG fall back together across a Spark shuffle` catches it. Grouped AVG already exports `(0.0, 0)` and stays correct. So on main this now costs coverage, not correctness: AVG partials, along with the other aggregates in the same operator, run in Spark whenever the final does. #5420 fixes the state and lets nondecimal AVG partials feed a Spark final again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org