You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/13 15:41:39 UTC
[PR] perf: optimize spark_arrays_overlap in datafusion-comet-spark-expr [datafusion-comet]
andygrove opened a new pull request, #4906:
URL: https://github.com/apache/datafusion-comet/pull/4906
> This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.
## Which issue does this PR close?
N/A — autonomous exploratory PR.
## Rationale for this change
Replaced the per-probe-element Arrow eq kernel call (which allocated a BooleanArray per element) with typed scans over the flat values buffers plus a hash probe for large lists, cutting allocations and kernel dispatches to zero on flat element types.
## What changes are included in this PR?
Replaced the per-probe-element Arrow eq kernel call (which allocated a BooleanArray per element) with typed scans over the flat values buffers plus a hash probe for large lists, cutting allocations and kernel dispatches to zero on flat element types.
## How are these changes tested?
Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs `main`).
Benchmark (criterion):
- spark_arrays_overlap_ int32 medium lists: 92.925% faster (base 11690342ns -> cand 827066ns)
- spark_arrays_overlap_ utf8 short lists: 77.473% faster (base 3281623ns -> cand 739251ns)
- spark_arrays_overlap_ utf8 long lists: 94.733% faster (base 11544605ns -> cand 608100ns)
- spark_arrays_overlap_ int32 long lists: 94.947% faster (base 12375951ns -> cand 625320ns)
- spark_arrays_overlap_ int32 short lists: 94.474% faster (base 3018371ns -> cand 166793ns)
Full criterion output:
```text
spark_arrays_overlap: int32 short lists
time: [166.19 µs 166.36 µs 166.55 µs]
change: [−94.489% −94.474% −94.460%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe
spark_arrays_overlap: int32 medium lists
time: [823.79 µs 825.75 µs 827.78 µs]
change: [−92.947% −92.925% −92.903%] (p = 0.00 < 0.05)
Performance has improved.
spark_arrays_overlap: int32 long lists
time: [623.94 µs 625.77 µs 627.46 µs]
change: [−94.965% −94.947% −94.930%] (p = 0.00 < 0.05)
Performance has improved.
spark_arrays_overlap: utf8 short lists
time: [728.82 µs 731.53 µs 734.73 µs]
change: [−77.607% −77.473% −77.335%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
spark_arrays_overlap: utf8 long lists
time: [608.52 µs 609.75 µs 610.95 µs]
change: [−94.755% −94.733% −94.712%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) low mild
1 (1.00%) high mild
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: optimize `spark_arrays_overlap` (up to 18x faster) [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4906:
URL: https://github.com/apache/datafusion-comet/pull/4906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org