You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/10/04 17:31:26 UTC
[I] Specialize Boolean interleave for scattered selections [arrow-rs]
sunchao opened a new issue, #11379:
URL: https://github.com/apache/arrow-rs/issues/11379
### Problem description
`arrow_select::interleave::interleave` has specialized paths for primitive, byte, view, dictionary and nested arrays, while Boolean arrays use the `MutableArrayData` fallback. For scattered row selections this repeats descriptor/bookkeeping work around individual bitmap bits.
Comet PR https://github.com/apache/datafusion-comet/pull/6430 has a top-level Boolean specialization using `BooleanBuffer::collect_bool` for values and validity. The reviewer requested moving the optimization upstream so nested Booleans also benefit through recursive interleave, and Comet can remove its copy after upgrading Arrow.
### Proposed improvement
Add a Boolean specialization in arrow-select. Preserve sliced offsets, nulls, row ordering and existing bounds/error behavior. Compare scattered selections with contiguous/clustered runs before choosing dispatch, since the current fallback efficiently copies bitmap ranges. Include non-byte-aligned slices, nullable/all-null arrays, short and empty selections, and nested-array coverage.
The Comet discussion includes Arrow 59.3.0 kernel and end-to-end shuffle measurements from both the author and reviewer. Those results motivate the change but are not a benchmark of current arrow-rs main. I checked the current main dispatch while filing this issue and Boolean still selects the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Specialize Boolean interleave for scattered selections [arrow-rs]
Posted by "dhruvtnwr28 (via GitHub)" <gi...@apache.org>.
dhruvtnwr28 commented on issue #11379:
URL: https://github.com/apache/arrow-rs/issues/11379#issuecomment-6098985219
Hi! I'm interested in investigating this issue as a contributor.
I'll start by understanding the current Boolean fallback in `arrow-select`, reviewing the existing interleave tests, and studying the proposed specialization and its correctness requirements.
I'm still getting familiar with the arrow-rs codebase, so I'll first validate the current behavior and test setup before committing to an implementation. If there are any specific areas or existing tests you'd recommend starting with, I'd appreciate the guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Specialize Boolean interleave for scattered selections [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #11379: Specialize Boolean interleave for scattered selections
URL: https://github.com/apache/arrow-rs/issues/11379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org