You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/07/31 16:13:05 UTC
[PR] Use arrow make_comparator for nested structural equality in arrays_overlap and array_position [datafusion-comet]
peterxcli opened a new pull request, #5176:
URL: https://github.com/apache/datafusion-comet/pull/5176
## Which issue does this PR close?
Closes #5101.
## Rationale for this change
The nested fallback paths in `arrays_overlap` and `array_position` duplicate comparison behavior already provided by Arrow. `arrays_overlap` recursively compares nested elements and dispatches an equality kernel for each leaf pair, while `array_position` materializes a `ScalarValue` for every candidate element. Arrow's reusable `make_comparator` supports the same nested structural comparison without this per-element dispatch and allocation overhead.
## What changes are included in this PR?
- Build and reuse an Arrow comparator for nested `arrays_overlap` probe/search pairs, while retaining the existing flat fast paths and outer null handling.
- Build one comparator for the flattened values and search arrays in `array_position` instead of allocating `ScalarValue`s in the inner loop.
- Remove the hand-written list/struct structural equality helpers.
- Add focused coverage for nested NaN, signed-zero, and inner-null comparisons. The tests preserve the existing Arrow/`ScalarValue` total-order behavior.
## How are these changes tested?
- `cargo test -p datafusion-comet-spark-expr --lib` (568 passed)
- `cargo clippy -p datafusion-comet-spark-expr --lib --tests -- -D warnings`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5176:
URL: https://github.com/apache/datafusion-comet/pull/5176#issuecomment-5151924644
@andygrove thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5176:
URL: https://github.com/apache/datafusion-comet/pull/5176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: use arrow make_comparator for nested structural equality in arrays_overlap and array_position [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5176:
URL: https://github.com/apache/datafusion-comet/pull/5176#issuecomment-5159625130
@andygrove I only create the issue per your review but forgot to push the code change for other review comments, opened another https://github.com/apache/datafusion-comet/pull/5194 as follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org