You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/28 20:34:45 UTC
[PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
sunchao opened a new pull request, #58390:
URL: https://github.com/apache/spark/pull/58390
### What changes were proposed in this pull request?
Backports [#53695](https://github.com/apache/spark/pull/53695), commit `a4aa4adf4fd0f790d7b71b322292050f6cf8db38`, to `branch-4.0` for [SPARK-54918](https://issues.apache.org/jira/browse/SPARK-54918).
Extends `NormalizeFloatingNumbers` to `array_distinct`, `array_union`, `array_intersect`, `array_except`, and `arrays_overlap`. Runs normalization before eager expression evaluation while preserving the existing late optimizer pass.
The only backport adaptation omits `MAP_FROM_ENTRIES` from the combined import because that tree pattern does not exist on this branch. The fix and regression tests are otherwise unchanged.
### Why are the changes needed?
Floating-point arrays still distinguish `-0.0` from `0.0` in hash-based set operations on this maintenance branch. This can retain duplicate zeros, miss intersections, or return an incorrect set difference.
### Does this PR introduce _any_ user-facing change?
Yes. The affected array operations now normalize floating-point inputs consistently with Spark SQL equality semantics. For example, `array_distinct` over the DOUBLE array `[-0.0, 0.0, 1.0]` returns `[0.0, 1.0]` instead of retaining both zeros.
### How was this patch tested?
Preserves the upstream regression coverage: 10 optimizer tests and 11 DataFrame tests, including idempotence, literal and runtime inputs, inline tables, NaNs, and bit-level verification that the surviving zero is positive.
Native SBT validation on this maintenance branch with JDK 17:
- `NormalizeFloatingPointNumbersSuite`: **18 passed**, 0 failures.
- `DataFrameFunctionsSuite`, filtered to `SPARK-54918`: **11 passed**, 0 failures.
- `git diff --check` passed.
```sh
build/sbt \
'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingPointNumbersSuite' \
'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z SPARK-54918'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (codex-cli 0.151.0-alpha.9)
Codex assisted with backport preparation and validation. The original implementation and author attribution from #53695 are preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #58390:
URL: https://github.com/apache/spark/pull/58390#issuecomment-5458023821
@dongjoon-hyun do you think we should backport this to 3.5 too? since it is correctness related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed pull request #58390: [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations
URL: https://github.com/apache/spark/pull/58390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #58390:
URL: https://github.com/apache/spark/pull/58390#issuecomment-5458795143
Merged to `branch-4.0`. Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #58390:
URL: https://github.com/apache/spark/pull/58390#issuecomment-5458025191
thanks for the quick review btw!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54918][SQL][4.0] Normalize floating numbers in array set operations [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58390:
URL: https://github.com/apache/spark/pull/58390#issuecomment-5458034947
> do you think we should backport this to 3.5 too? since it is correctness related.
No, `branch-3.5` accepts only security-related patches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org