You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/13 15:41:39 UTC

[PR] perf: optimize spark_arrays_overlap in datafusion-comet-spark-expr [datafusion-comet]

andygrove opened a new pull request, #4906:
URL: https://github.com/apache/datafusion-comet/pull/4906

   > This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.
   
   
   ## Which issue does this PR close?
   
   N/A — autonomous exploratory PR.
   
   ## Rationale for this change
   
   Replaced the per-probe-element Arrow eq kernel call (which allocated a BooleanArray per element) with typed scans over the flat values buffers plus a hash probe for large lists, cutting allocations and kernel dispatches to zero on flat element types.
   
   ## What changes are included in this PR?
   
   Replaced the per-probe-element Arrow eq kernel call (which allocated a BooleanArray per element) with typed scans over the flat values buffers plus a hash probe for large lists, cutting allocations and kernel dispatches to zero on flat element types.
   
   ## How are these changes tested?
   
   Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs `main`).
   
   Benchmark (criterion):
   
   - spark_arrays_overlap_ int32 medium lists: 92.925% faster (base 11690342ns -> cand 827066ns)
   - spark_arrays_overlap_ utf8 short lists: 77.473% faster (base 3281623ns -> cand 739251ns)
   - spark_arrays_overlap_ utf8 long lists: 94.733% faster (base 11544605ns -> cand 608100ns)
   - spark_arrays_overlap_ int32 long lists: 94.947% faster (base 12375951ns -> cand 625320ns)
   - spark_arrays_overlap_ int32 short lists: 94.474% faster (base 3018371ns -> cand 166793ns)
   
   Full criterion output:
   
   ```text
   spark_arrays_overlap: int32 short lists
                           time:   [166.19 µs 166.36 µs 166.55 µs]
                           change: [−94.489% −94.474% −94.460%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 3 outliers among 100 measurements (3.00%)
     1 (1.00%) low mild
     1 (1.00%) high mild
     1 (1.00%) high severe
   
   spark_arrays_overlap: int32 medium lists
                           time:   [823.79 µs 825.75 µs 827.78 µs]
                           change: [−92.947% −92.925% −92.903%] (p = 0.00 < 0.05)
                           Performance has improved.
   
   spark_arrays_overlap: int32 long lists
                           time:   [623.94 µs 625.77 µs 627.46 µs]
                           change: [−94.965% −94.947% −94.930%] (p = 0.00 < 0.05)
                           Performance has improved.
   
   spark_arrays_overlap: utf8 short lists
                           time:   [728.82 µs 731.53 µs 734.73 µs]
                           change: [−77.607% −77.473% −77.335%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 1 outliers among 100 measurements (1.00%)
     1 (1.00%) high mild
   
   spark_arrays_overlap: utf8 long lists
                           time:   [608.52 µs 609.75 µs 610.95 µs]
                           change: [−94.755% −94.733% −94.712%] (p = 0.00 < 0.05)
                           Performance has improved.
   Found 4 outliers among 100 measurements (4.00%)
     3 (3.00%) low mild
     1 (1.00%) high mild
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: optimize `spark_arrays_overlap` (up to 18x faster) [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4906:
URL: https://github.com/apache/datafusion-comet/pull/4906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org