You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/08/01 03:13:48 UTC
[I] Nested array comparison does not match Spark for signed zero [datafusion-comet]
peterxcli opened a new issue, #5191:
URL: https://github.com/apache/datafusion-comet/issues/5191
## Describe the bug
Nested floating-point elements use total-order comparison in Comet, where `-0.0` and `0.0` are distinct. Spark uses `ordering.equiv` for nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.
This affects:
- `arrays_overlap` when the array elements are themselves arrays or structs containing floats.
- The nested fallback in `array_position`.
The flat `arrays_overlap` path is intentionally different: Spark uses `Double.equals` there and distinguishes signed zeros.
## Steps to reproduce
```sql
CREATE TABLE t(
a ARRAY<ARRAY<DOUBLE>>,
b ARRAY<ARRAY<DOUBLE>>,
v ARRAY<DOUBLE>
) USING parquet;
INSERT INTO t VALUES (
array(array(0.0D)),
array(array(-0.0D)),
array(-0.0D)
);
SELECT arrays_overlap(a, b), array_position(a, v) FROM t;
```
Spark returns `true` and `1`; Comet returns `false` and `0`.
## Expected behavior
Nested `arrays_overlap` and `array_position` should treat `-0.0` as equal to `0.0`, matching Spark, while continuing to treat NaN as equal to NaN.
## Additional context
Tracked separately from #5176 because the mismatch predates that refactor. Arrow's `make_comparator` uses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Nested array comparison does not match Spark for signed zero [datafusion-comet]
Posted by "divyankshah (via GitHub)" <gi...@apache.org>.
divyankshah commented on issue #5191:
URL: https://github.com/apache/datafusion-comet/issues/5191#issuecomment-5166265618
Hi, I'd like to work on this if it's not claimed yet.
My Approach is to normalize -0.0 to 0.0 in the nested float leaves before building the comparator (the arrays_overlap nested path and array_position's nested fallback), leaving NaN handling as-is since that already matches Spark. Will also add proper regression test based on the repro above.
I have observed #5194 touches the same comparator code (perf hoist, different issue). Will rebase on top of whichever merges first.
Will open a PR soon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Nested array comparison does not match Spark for signed zero [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5191: Nested array comparison does not match Spark for signed zero
URL: https://github.com/apache/datafusion-comet/issues/5191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org