You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/08/02 18:47:38 UTC
Re: [PR] fix: normalize floating-point values in native collect_set [datafusion-comet]
peterxcli commented on PR #5166:
URL: https://github.com/apache/datafusion-comet/pull/5166#issuecomment-5159819008
@andygrove thanks for the review! review change is pushed, issue is opened per your review. please take another look, thanks!
> 1. Gate incompatibility reasons by Spark version
`getIncompatibleReasons()` now returns `Nil` on Spark 4.2+, while retaining the compatibility warning for Spark 3.4–4.1.
> 2. Normalize `Complete` aggregates
Floating-point normalization now applies to both `Partial` and `Complete` modes. Merge modes remain unchanged because they consume aggregate buffers rather than raw rows.
> 3. Assert collected values
The regression queries now compare `sort_array(collect_set(...))` results instead of only checking collection sizes.
> 4. Restore scalar coverage
Added float and double cases covering ordinary values, NULL, NaN, positive/negative infinity, and signed zero. Tests run with Parquet dictionary encoding both enabled and disabled.
> 5. Expand nested-value coverage
Added null structs, null arrays, arrays containing null elements, `array<struct<double>>`, and `struct<array<double>>`, including NaN and signed-zero values.
> 6. Test global aggregation
Added a repartitioned `collect_set` query without `GROUP BY` to exercise partial-buffer merging into a global aggregate.
> 7. Check Spark 4.2 SQL-test diffs
Confirmed that no `dev/diffs/4.2*.diff` or Spark 4.2 Spark SQL Tests job currently exists. No diff file was modified.
> 8. Document both pre-4.2 divergences
The compatibility messages now explain that Comet deduplicates NaNs but distinguishes `-0.0` from `0.0`, while pre-4.2 Spark does the opposite. Negative-zero fixtures now use string casts so the sign is preserved.
> 9. File a Spark 4.2 tracking issue
Filed https://github.com/apache/datafusion-comet/issues/5209 to enable the two SPARK-57298 `DataFrameAggregateSuite` tests once the Spark 4.2 diff and CI job are added.
> 10. Keep the original compatibility wording
Restored the original “Before Spark 4.2…” wording and added only the missing signed-zero explanation.
> 11. Remove the mode explanation comment
Removed the added explanatory comment above `normalizeFloatingNumbers`; the `Partial | Complete` behavior remains unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org