You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mightsleep (via GitHub)" <gi...@apache.org> on 2026/10/07 23:12:28 UTC
[I] Speed up merge with a scalar side [arrow-rs]
mightsleep opened a new issue, #11437:
URL: https://github.com/apache/arrow-rs/issues/11437
### Is your feature request related to a problem or challenge?
DataFusion evaluates CASE WHEN c THEN expr ELSE literal END with kernels::merge, the scalar on one side. merge extends MutableArrayData once per row for the scalar side (try_extend(i, 0, 1) in arrow-select/src/merge.rs), about 6 ns per row.
### Describe the solution you'd like
With a scalar side, merge is the transpose of filter: fill the output with the scalar and write the array side to the mask's set bits (set_indices). The validity has closed forms: no null buffer when neither side has nulls, the mask itself for a NULL scalar, otherwise (scalar valid ? !mask : 0) | expand(array validity, mask) in the same loop. Utf8View works the same way on the views, sharing the array's buffers. DataFusion's own scatter (physical-expr-common/src/utils.rs) already does this for a CASE without ELSE.
Prototype numbers:
- kernel, TPC-H SF1 CASE inputs (Q14, Q8, Decimal128): 6 ns to 0.2 ns per row
- DataFusion main, 10M ClickBench rows, one thread: SUM(CASE WHEN "IsMobile" = 1 THEN "ResolutionWidth" ELSE 0 END) 90 ms to 42 ms, COUNT(DISTINCT CASE ... ELSE 'desktop' END) 114 ms to 58 ms
- dense masks with long runs gain little; there the existing run copy stays
I plan to open a PR for primitive and view types, benchmarks first.
### Describe alternatives you've considered
Interleave with running counters, as zip does since #10368: about 2.6 ns per row and slower than today on long runs.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Speed up merge with a scalar side [arrow-rs]
Posted by "mightsleep (via GitHub)" <gi...@apache.org>.
mightsleep commented on issue #11437:
URL: https://github.com/apache/arrow-rs/issues/11437#issuecomment-6171072640
Adding zip with a scalar side to the plan: same per-row MutableArrayData cost (6 to 7 ns a row), and DataFusion could zip the full column for an infallible THEN instead of filter + merge. PR coming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Speed up merge with a scalar side [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on issue #11437:
URL: https://github.com/apache/arrow-rs/issues/11437#issuecomment-6125820894
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Speed up merge with a scalar side [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on issue #11437:
URL: https://github.com/apache/arrow-rs/issues/11437#issuecomment-6146688567
> 6\. Yes hello [@Abhisheklearn12](https://github.com/Abhisheklearn12) thanks for the interest. I opened this one with the plan and prototypes above and I'm working through the stacked PRs (benchmarks are up in [#11527](https://github.com/apache/arrow-rs/pull/11527), then primitives and decimals, Boolean, Utf8View and Utf8) rn, so I'd like to keep those. merge_n isn't designed yet and would be a good separate piece if you'd like to take it. Would that work for you?
Thanks @mightsleep, that works for me! Very sorry, I didn't see you were already working through the stack. I'll leave those to you. I'll take merge_n, before writing code, I'll put together a quick benchmark to see how much there is to gain when some inputs are literals, and post that here along with a rough design. I'll build on your scatter changes once they're up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Speed up merge with a scalar side [arrow-rs]
Posted by "mightsleep (via GitHub)" <gi...@apache.org>.
mightsleep commented on issue #11437:
URL: https://github.com/apache/arrow-rs/issues/11437#issuecomment-6130303528
Yes hello @Abhisheklearn12 thanks for the interest. I opened this one with the plan and prototypes above and I'm working through the stacked PRs (benchmarks are up in #11527, then primitives and decimals, Boolean, Utf8View and Utf8) rn, so I'd like to keep those. merge_n isn't designed yet and would be a good separate piece if you'd like to take it. Would that work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org