You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2025/09/24 10:13:43 UTC
[I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
wForget opened a new issue, #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448
### Describe the bug
test case:
```
test("SUM, MIN, MAX, AVG for NaN, -0.0 and 0.0") {
// NaN should be grouped together, and -0.0 and 0.0 should also be grouped together
Seq(true, false).foreach { dictionaryEnabled =>
val data: Seq[(Float, Int)] = Seq(
(Float.NaN, 1),
(-0.0.asInstanceOf[Float], 2),
(0.0.asInstanceOf[Float], 3),
(Float.NaN, 4))
withSQLConf(CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true") {
withParquetTable(data, "tbl", dictionaryEnabled) {
checkSparkAnswer("SELECT SUM(_2), MIN(_2), MAX(_2), _1 FROM tbl GROUP BY _1")
checkSparkAnswer("SELECT MIN(_1), MAX(_1), MIN(_2), MAX(_2) FROM tbl")
checkSparkAnswer("SELECT AVG(_2), _1 FROM tbl GROUP BY _1")
checkSparkAnswer("SELECT AVG(_1), AVG(_2) FROM tbl")
}
}
}
}
```
error:
```
== Results ==
!== Correct Answer - 1 == == Spark Answer - 1 ==
struct<min(_1):float,max(_1):float,min(_2):int,max(_2):int> struct<min(_1):float,max(_1):float,min(_2):int,max(_2):int>
![0.0,NaN,1,4] [-0.0,NaN,1,4]
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3346302963
> We might be able to simply fix this by wrapping NormalizeNaNAndZero in the min/max agg for float types in comet.
This may not be a solution. Spark only does normalization when sorting, so returning `-0.0` or `0.0` is unstable.
```
test("test min on -0.0 and 0.0") {
withSQLConf(
CometConf.COMET_SPARK_TO_ARROW_ENABLED.key -> "true",
CometConf.COMET_SPARK_TO_ARROW_SUPPORTED_OPERATOR_LIST.key -> "LocalTableScan",
CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true"
) {
// succeed with -0.0
checkSparkAnswer("SELECT min(col1) FROM VALUES (-0.0f), (0.0f)")
// failed, spark returns 0.0, comet returns -0.0
checkSparkAnswer("SELECT min(col1) FROM VALUES (0.0f), (-0.0f)")
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3347730439
> This may not be a solution. Spark only does normalization when sorting, so returning -0.0 or 0.0 is unstable.
```
checkSparkAnswer("select min(_1) from (SELECT _1 FROM tbl order by _1 desc)")
```
passes
whereas
```
checkSparkAnswer("select min(_1) from (SELECT _1 FROM tbl order by _1 asc)")
```
I'll check little bit more today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3329564561
Thanks @wForget
might be related to https://github.com/apache/datafusion/issues/11051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3342165077
> I dont see normalized code is called, checking
https://github.com/apache/datafusion-comet/blob/03c062646ba183069b2fd252df23d00e1f2e51c8/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L575-L578
As mentioned in the comment, `SQLOrderingUtil.compareFloats` is used in vanilla Spark to compare float types without normalization.
<img width="1485" height="779" alt="Image" src="https://github.com/user-attachments/assets/4cd69479-7624-4069-ad9e-ce4f78b183ce" />
We might be able to simply fix this by wrapping `NormalizeNaNAndZero` in the `min/max` agg for float types in comet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2448: Mismatched `min` value for -0.0 and 0.0
URL: https://github.com/apache/datafusion-comet/issues/2448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3338510141
Possibly relevant code:
https://github.com/apache/datafusion-comet/blob/03c062646ba183069b2fd252df23d00e1f2e51c8/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L600-L612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3339132669
made a quick investigation, it fails on MIN if shuffle enabled. Let me check if quick fix available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Mismatched `min` value for -0.0 and 0.0 [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2448:
URL: https://github.com/apache/datafusion-comet/issues/2448#issuecomment-3341996202
I dont see normalized code is called, checking
but Spark and DF has diff vision on min
```
scala> spark.sql("select min(a) from (select cast(-0.0 as float) a union all select cast(0.0 as float))").show(false)
+------+
|min(a)|
+------+
|0.0 |
+------+
```
```
> select min(a) from (select 0.0::float a union all select -0.0::float);
+--------+
| min(a) |
+--------+
| -0.0 |
+--------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org