You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/09/22 18:55:04 UTC
[PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Rich-T-kid opened a new pull request, #11174:
URL: https://github.com/apache/arrow-rs/pull/11174
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Closes #11141 & #8957
# Rationale for this change
Callers that need to interleave rows from multiple source batches (top-K emission, hash join materialisation, sort-merge join) currently must:
1. Call interleave_record_batch → allocates an intermediate RecordBatch
2. Call push_batch on the result → copies again into the coalescer
3. Drop the intermediate batch
push_batch_interleaved collapses both steps into one call, (possibly) removing the intermediate allocation.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
This PR exists only to establish the API. The implementation is intentionally equivalent to calling interleave_record_batch + push_batch. A follow-up PR, guided by the benchmarks also introduced here, will optimize the path by avoiding the intermediate RecordBatch allocation.
Benchmarks
Added interleave group to coalesce_kernels covering the same null_density × selectivity matrix as the existing filter/take groups, across:
- primitive (int32 / float64 / timestamp) — specialized path
- single_utf8view at two string lengths — exercises inline vs buffer split
- mixed_fsb (16+32) — FixedSizeBinary specialized path
- mixed_generic (binary/bool/dict) — generic path
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are these changes tested?
yes test are included.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
# Are there any user-facing changes?
new public API method on `BatchCoalescer`
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6293432032
@mbutrovich thank you for the reviews. there are regressions for non targeted cases but overall benchmark results are still strong. these issues https://github.com/apache/arrow-rs/issues/11738, https://github.com/apache/arrow-rs/issues/11720, https://github.com/apache/arrow-rs/issues/11719, https://github.com/apache/arrow-rs/issues/11718, https://github.com/apache/arrow-rs/issues/11563, https://github.com/apache/arrow-rs/issues/11567 should address them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179337553
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6173013040
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6171978547)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (ab144b34f62925a5f33f13a13e3fc8b9f840d15f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..ab144b34f62925a5f33f13a13e3fc8b9f840d15f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.86 13.6±0.15ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.61 3.4±0.02ms ? ?/sec 1.00 941.6±1.93µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 4.12 3.2±0.12ms ? ?/sec 1.00 775.4±1.18µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 5.13 2.3±0.13ms ? ?/sec 1.00 442.2±0.73µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 4.70 20.6±0.09ms ? ?/sec 1.00 4.4±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 2.01 4.6±0.02ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.61 3.4±0.17ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 864.0±2.82µs ? ?/sec 1.97 1702.9±5.49µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.48 12.7±0.10ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.70 2.3±0.02ms ? ?/sec 1.00 857.9±4.06µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.16 1637.8±5.23µs ? ?/sec 1.00 758.4±0.73µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.54 1111.2±4.78µs ? ?/sec 1.00 437.5±1.17µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 5.33 19.7±0.09ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.59 3.6±0.02ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1989.5±6.53µs ? ?/sec 1.06 2.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 854.0±2.39µs ? ?/sec 1.97 1685.9±6.10µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 114.7±5.30ms ? ?/sec 1.03 118.4±1.69ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.4±0.05ms ? ?/sec 1.00 5.4±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.02 3.8±0.03ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 111.1±7.09ms ? ?/sec 1.03 114.2±3.78ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.9±0.04ms ? ?/sec 1.01 5.9±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.03ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.02 2.1±0.03ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 175.4±4.29ms ? ?/sec 1.01 176.7±5.62ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.01 23.0±0.15ms ? ?/sec 1.00 22.8±0.08ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.01 6.7±0.07ms ? ?/sec 1.00 6.7±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 6.3±0.48ms ? ?/sec 1.06 6.7±0.27ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 6.1±0.31ms ? ?/sec 1.04 6.3±0.22ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.01 33.4±0.24ms ? ?/sec 1.00 33.0±0.25ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.3±0.08ms ? ?/sec 1.02 8.4±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.4±0.24ms ? ?/sec 1.01 6.5±0.33ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 5.2±0.48ms ? ?/sec 1.01 5.3±0.44ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.91 13.6±0.07ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.50 3.3±0.02ms ? ?/sec 1.00 943.2±3.21µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.80 2.9±0.16ms ? ?/sec 1.00 765.4±0.89µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.79 2.1±0.11ms ? ?/sec 1.00 444.8±1.53µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 4.75 20.6±0.08ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 2.01 4.5±0.04ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.58 3.3±0.13ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 860.7±1.90µs ? ?/sec 1.97 1695.5±6.02µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.87 12.7±0.08ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.73 2.3±0.02ms ? ?/sec 1.00 835.1±3.48µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.80 1350.5±4.45µs ? ?/sec 1.00 752.1±1.00µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.31 574.5±1.46µs ? ?/sec 1.00 438.5±1.03µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 5.41 19.7±0.09ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.60 3.4±0.02ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1732.5±4.46µs ? ?/sec 1.19 2.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 859.4±2.15µs ? ?/sec 1.96 1685.4±5.23µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 14.84 12.3±0.47ms ? ?/sec 1.00 828.9±5.38µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.43 30.4±0.32ms ? ?/sec 1.00 4.7±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.10 4.7±0.03ms ? ?/sec 1.00 1523.2±10.43µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.52 1737.2±3.98µs ? ?/sec 1.00 1144.6±1.42µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.15 844.0±0.92µs ? ?/sec 1.00 737.1±0.97µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 6.33 47.3±0.27ms ? ?/sec 1.00 7.5±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.28 7.0±0.04ms ? ?/sec 1.00 5.4±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.64 4.4±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1524.5±2.73µs ? ?/sec 2.48 3.8±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.6±0.15ms ? ?/sec 1.02 24.0±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.01ms ? ?/sec 1.03 5.3±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.9±0.00ms ? ?/sec 1.01 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.01 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.15ms ? ?/sec 1.01 33.1±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.3±0.05ms ? ?/sec 1.09 6.9±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.04 3.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.65 13.4±0.18ms ? ?/sec 1.00 5.1±0.05ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.87 3.0±0.03ms ? ?/sec 1.00 1054.1±4.26µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.48 3.2±0.01ms ? ?/sec 1.00 917.4±1.32µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.31 828.6±1.00µs ? ?/sec 1.00 633.2±0.56µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 3.46 20.2±0.08ms ? ?/sec 1.00 5.8±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 2.04 4.2±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.75 3.5±0.01ms ? ?/sec 1.00 1990.8±4.25µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1066.5±1.66µs ? ?/sec 1.60 1705.4±4.31µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.07ms ? ?/sec 1.02 13.8±0.05ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1552.4±9.08µs ? ?/sec 1.01 1572.3±5.25µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 360.5±0.75µs ? ?/sec 1.01 363.9±0.47µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.9±0.26µs ? ?/sec 1.01 243.0±0.33µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.3±0.16ms ? ?/sec 1.00 20.2±0.05ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 616.5±2.07µs ? ?/sec 1.00 619.3±0.75µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.6±0.52µs ? ?/sec 1.00 431.8±0.44µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.12ms ? ?/sec 1.02 12.7±0.06ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.02 1735.1±12.42µs ? ?/sec 1.00 1700.2±11.20µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.05 751.1±1.48µs ? ?/sec 1.00 717.6±1.45µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.03 471.0±1.13µs ? ?/sec 1.00 456.9±0.64µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.01 19.6±0.26ms ? ?/sec 1.00 19.4±0.13ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.02ms ? ?/sec 1.02 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1073.8±3.26µs ? ?/sec 1.02 1090.1±2.71µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 701.1±1.79µs ? ?/sec 1.01 709.3±0.99µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.2±0.16ms ? ?/sec 1.00 13.3±0.03ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.02ms ? ?/sec 1.05 2.5±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 2.0±0.01ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1509.7±4.10µs ? ?/sec 1.01 1525.1±5.73µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.01 21.0±0.19ms ? ?/sec 1.00 20.7±0.08ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.02ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.01ms ? ?/sec 1.05 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1742.8±5.65µs ? ?/sec 1.04 1805.1±6.03µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.8±0.31ms ? ?/sec 1.01 35.2±0.32ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.03ms ? ?/sec 1.01 5.1±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.03 1996.3±3.55µs ? ?/sec 1.00 1935.9±5.88µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.01 1219.8±2.14µs ? ?/sec 1.00 1202.4±2.58µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.2±0.21ms ? ?/sec 1.03 59.9±0.29ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.03ms ? ?/sec 1.01 11.0±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.3±0.01ms ? ?/sec 1.01 5.3±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.0±0.30ms ? ?/sec 1.02 45.0±0.11ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.3±0.21ms ? ?/sec 1.07 9.9±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.02ms ? ?/sec 1.03 5.3±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.8±0.25ms ? ?/sec 1.01 52.4±0.14ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.11ms ? ?/sec 1.05 11.2±0.31ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.1±0.02ms ? ?/sec 1.04 6.3±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.5±0.04ms ? ?/sec 1.00 4.5±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.1±0.17ms ? ?/sec 1.01 36.4±0.22ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.8±0.09ms ? ?/sec 1.00 12.7±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 14.3±0.73ms ? ?/sec 1.01 14.4±0.87ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.21 14.0±1.46ms ? ?/sec 1.00 11.6±1.72ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.9±0.35ms ? ?/sec 1.02 47.6±0.29ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.1±0.06ms ? ?/sec 1.05 14.8±0.09ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.10 14.0±0.51ms ? ?/sec 1.00 12.8±0.68ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.02 12.5±0.82ms ? ?/sec 1.00 12.2±0.71ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.76 13.4±0.19ms ? ?/sec 1.00 4.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.76 3.0±0.02ms ? ?/sec 1.00 1077.1±26.57µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 3.04 2.7±0.01ms ? ?/sec 1.00 885.5±3.22µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.31 829.0±1.31µs ? ?/sec 1.00 633.9±0.92µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 3.47 20.1±0.09ms ? ?/sec 1.00 5.8±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 2.04 4.1±0.02ms ? ?/sec 1.00 2.0±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.48 2.8±0.01ms ? ?/sec 1.00 1924.8±14.60µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1059.3±1.75µs ? ?/sec 1.60 1691.1±13.50µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1wzwczws-b8f8a7e3-njkg
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6171978547-3389-qq866 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1380.3s |
| Peak memory | 59.9 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1352.3s |
| CPU sys | 23.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1415.3s |
| Peak memory | 59.9 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1386.1s |
| CPU sys | 23.6s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6171978547:finish:aa7ec84e1620f7d843dcc2c18499e862 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6223362051
Maybe we can do a fail fast pass. Begin scanning the indices & if selectivity is very high we'll fall back immediately. Should probably do this in chunks of 1/8 or 1/16 the size of the indices so we can avoid extra comparisons in tight loops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6227332700
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226709936)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.23 13.8±0.12ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.72 3.4±0.02ms ? ?/sec 1.00 1981.2±13.90µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.01 3.1±0.16ms ? ?/sec 1.00 3.1±0.16ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.01 2.2±0.08ms ? ?/sec 1.00 2.2±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.5±0.09ms ? ?/sec 1.00 15.3±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.21 4.6±0.03ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.3±0.20ms ? ?/sec 1.14 3.8±0.09ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 863.5±1.20µs ? ?/sec 1.01 874.5±3.69µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.89 12.9±0.13ms ? ?/sec 1.00 2.6±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.06 2.3±0.02ms ? ?/sec 1.00 1138.8±13.13µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.06 1669.2±4.32µs ? ?/sec 1.00 1569.1±6.99µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.02 1126.2±5.71µs ? ?/sec 1.00 1099.3±7.33µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.36 19.7±0.09ms ? ?/sec 1.00 14.5±0.09ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.20 3.6±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1983.2±5.92µs ? ?/sec 1.05 2.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 857.8±1.31µs ? ?/sec 1.00 859.1±2.03µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.05 112.4±4.86ms ? ?/sec 1.00 106.8±5.56ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.03 5.6±0.03ms ? ?/sec 1.00 5.5±0.08ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.02 3.8±0.03ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.02 2.1±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 110.1±4.93ms ? ?/sec 1.02 112.3±0.95ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.03 5.9±0.04ms ? ?/sec 1.00 5.7±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.7±0.02ms ? ?/sec 1.00 3.6±0.06ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.03 2.1±0.06ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 172.4±5.00ms ? ?/sec 1.03 177.6±5.38ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.03 23.0±0.13ms ? ?/sec 1.00 22.3±0.13ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.03 6.8±0.07ms ? ?/sec 1.00 6.6±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.10 6.9±0.16ms ? ?/sec 1.00 6.3±0.45ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.01 6.1±0.21ms ? ?/sec 1.00 6.0±0.31ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.02 33.4±0.24ms ? ?/sec 1.00 32.6±0.24ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.05 8.5±0.08ms ? ?/sec 1.00 8.1±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.2±0.28ms ? ?/sec 1.03 6.3±0.19ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 5.6±0.20ms ? ?/sec 1.02 5.7±0.27ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.32 13.6±0.09ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.81 3.4±0.02ms ? ?/sec 1.00 1861.4±7.41µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.9±0.14ms ? ?/sec 1.09 3.2±0.27ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.01 2.2±0.09ms ? ?/sec 1.00 2.2±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.5±0.05ms ? ?/sec 1.00 15.3±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.24 4.6±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.3±0.10ms ? ?/sec 1.00 3.3±0.15ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 858.6±1.21µs ? ?/sec 1.01 867.7±3.65µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.05 13.0±0.12ms ? ?/sec 1.00 2.6±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.17 2.3±0.02ms ? ?/sec 1.00 1076.0±11.31µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.11 1390.5±9.77µs ? ?/sec 1.00 1249.1±2.77µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.04 592.3±1.11µs ? ?/sec 1.00 569.8±0.71µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.39 19.7±0.05ms ? ?/sec 1.00 14.1±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.21 3.5±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1778.9±8.72µs ? ?/sec 1.01 1801.8±4.64µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.01 862.4±1.64µs ? ?/sec 1.00 858.0±3.69µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.8±0.31ms ? ?/sec 1.08 12.7±1.12ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 5.84 30.3±0.48ms ? ?/sec 1.00 5.2±0.14ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.06 4.7±0.04ms ? ?/sec 1.00 1530.2±77.44µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.51 1749.1±15.01µs ? ?/sec 1.00 1157.7±3.88µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.02 833.0±0.94µs ? ?/sec 1.00 817.7±1.53µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.5±0.18ms ? ?/sec 1.00 33.6±0.19ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.26 7.2±0.19ms ? ?/sec 1.00 5.7±0.20ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.11 2.7±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1523.0±1.47µs ? ?/sec 1.02 1549.8±2.83µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.7±0.34ms ? ?/sec 1.00 23.8±0.42ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.03 5.3±0.04ms ? ?/sec 1.00 5.1±0.06ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.08 3.0±0.00ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.03 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.2±0.36ms ? ?/sec 1.01 33.4±0.45ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.09 6.9±0.11ms ? ?/sec 1.00 6.4±0.07ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.0±0.10ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.02 2.2±0.11ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.00 13.5±0.20ms ? ?/sec 1.00 4.5±0.11ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.45 3.0±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.3±0.00ms ? ?/sec 1.10 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.06 860.1±0.93µs ? ?/sec 1.00 810.6±1.19µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.32 20.1±0.10ms ? ?/sec 1.00 15.3±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.16 4.1±0.02ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.5±0.01ms ? ?/sec 1.15 4.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.02 1089.3±1.18µs ? ?/sec 1.00 1064.1±1.88µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.9±0.20ms ? ?/sec 1.00 13.9±0.20ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.02 1607.2±14.42µs ? ?/sec 1.00 1581.6±22.01µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 359.7±1.29µs ? ?/sec 1.01 362.6±1.59µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.3±0.28µs ? ?/sec 1.00 240.2±0.43µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.01 20.6±0.17ms ? ?/sec 1.00 20.5±0.29ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.01 2.6±0.01ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.01 623.0±1.25µs ? ?/sec 1.00 617.3±1.93µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 432.9±0.27µs ? ?/sec 1.00 433.4±0.49µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.7±0.11ms ? ?/sec 1.01 12.8±0.17ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1685.0±19.11µs ? ?/sec 1.04 1760.4±16.19µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.08 758.7±1.38µs ? ?/sec 1.00 701.2±2.23µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.06 477.8±0.62µs ? ?/sec 1.00 451.6±0.62µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.4±0.17ms ? ?/sec 1.00 19.4±0.16ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.02ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.01 1053.3±3.04µs ? ?/sec 1.00 1047.9±1.82µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 688.5±0.89µs ? ?/sec 1.02 703.3±0.87µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.1±0.05ms ? ?/sec 1.02 13.4±0.17ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.10 2.1±0.00ms ? ?/sec 1.00 1905.8±5.70µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.08 1582.6±7.65µs ? ?/sec 1.00 1464.5±6.75µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.9±0.19ms ? ?/sec 1.01 21.0±0.15ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.03ms ? ?/sec 1.00 3.9±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.01ms ? ?/sec 1.02 2.4±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1730.6±4.44µs ? ?/sec 1.02 1757.8±12.81µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 35.0±0.27ms ? ?/sec 1.01 35.2±0.24ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.03ms ? ?/sec 1.02 5.1±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.06 2.0±0.01ms ? ?/sec 1.00 1905.2±4.54µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.03 1244.4±6.43µs ? ?/sec 1.00 1211.1±2.25µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.9±0.15ms ? ?/sec 1.01 59.2±0.17ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.01 10.8±0.03ms ? ?/sec 1.00 10.7±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.1±0.01ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.01ms ? ?/sec 1.00 4.4±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.01 44.9±0.26ms ? ?/sec 1.00 44.6±0.28ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.07 9.7±0.20ms ? ?/sec 1.00 9.0±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.07 5.4±0.01ms ? ?/sec 1.00 5.0±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.03 3.8±0.01ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.3±0.36ms ? ?/sec 1.00 52.1±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.07 11.2±0.34ms ? ?/sec 1.00 10.5±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.9±0.02ms ? ?/sec 1.00 5.9±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.5±0.03ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.02 36.7±0.23ms ? ?/sec 1.00 36.0±0.22ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.04 13.0±0.07ms ? ?/sec 1.00 12.4±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 14.0±0.69ms ? ?/sec 1.02 14.3±0.54ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 13.4±1.20ms ? ?/sec 1.10 14.7±0.47ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.01 47.7±0.23ms ? ?/sec 1.00 47.4±0.23ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.3±0.08ms ? ?/sec 1.00 14.2±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.06 12.8±0.67ms ? ?/sec 1.00 12.1±0.89ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 12.5±0.53ms ? ?/sec 1.04 13.0±0.79ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.98 13.2±0.14ms ? ?/sec 1.00 4.4±0.11ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.59 3.0±0.01ms ? ?/sec 1.00 1858.8±6.56µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.00ms ? ?/sec 1.05 2.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.05 859.4±0.84µs ? ?/sec 1.00 816.4±1.42µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.37 20.1±0.09ms ? ?/sec 1.00 14.8±0.07ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.25 4.1±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.00ms ? ?/sec 1.13 3.3±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.02 1091.6±1.27µs ? ?/sec 1.00 1067.5±1.49µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-149lv8yb-4859eb8f-ptpz
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6226709936-3530-7l4tm 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1380.3s |
| Peak memory | 60.4 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1352.1s |
| CPU sys | 22.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.2 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1367.6s |
| CPU sys | 22.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6226709936:finish:3d56f650832c4bcc0472fe67f21cff5b -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225097447
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6224942383
## tldr; going to add a new method onto the batch Coacler
I thought about changing to this API (`push_batch_for_partition`, which takes a shared partition_ids array plus a target partition). The rationale is that we could save both the double row copy and the repeated index copies that happen for each output partition. In practice, though, each coalescer has to rescan all N ids to find its rows, so we’d be doing N*P work (~4M comparisons per 8K batch at 512 partitions).
At the per-record-batch scale, the work we’d be saving is small by comparison. The per-partition indices come to ~4 bytes per selected row (~32 KB total for an 8K batch). If those buffers are reused across batches, there are no repeated allocations and the whole set stays in cache. The double row copy is the expensive part, and the `push_batch_with_indices` hook already lets specialized impls skip it. The new API would trade a small cost for a multiplier that grows with P.
with that in mind, I do think we need a different api because once we convert the indices to a `dyn Array` we lose ownership of it, meaning we lose the backing array. we should be generic over the indices size type and aim for as small as possible units so we can make better use of CPU cache lines.
proposed api
```rust
push_batch_with_indices<T>(src: record_batch, indices: &[T]){}
```
~~~ indices can be Vec<u8>,...Vec<u64>. typically record batches will never need to represent anything larger than a u16 so we can pack cpu cache lines better, even more so for very sparse cases such as 64..512 output partitions. we'll also avoid the 2x memory cost of the alternative which is to create another vector of the same backing space before turning the original vector into a `dyn array`~~~
... I dont think we can do this in rust as it would require the entire struct to be generic.... a downcast should be fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6171980587
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6171978547)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (ab144b34f62925a5f33f13a13e3fc8b9f840d15f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..ab144b34f62925a5f33f13a13e3fc8b9f840d15f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6171978547:start:cf9a498e042d1d6f9815a1b10cb5ab63 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321680067
##########
arrow-select/src/coalesce.rs:
##########
@@ -2893,4 +3048,147 @@ mod tests {
"draining must release the accounted memory"
);
}
+
+ #[test]
+ fn test_push_batch_with_indices_splits_at_target_batch_size() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..1000i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from_iter_values((0..1000u32).step_by(3));
+ let target_batch_size = 100;
+
+ let mut coalescer = BatchCoalescer::new(Arc::clone(&schema), target_batch_size);
+ coalescer
+ .push_batch_with_indices(batch.clone(), &indices)
+ .unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ let mut completed = vec![];
+ while let Some(batch) = coalescer.next_completed_batch() {
+ completed.push(batch);
+ }
+ for batch in &completed[..completed.len() - 1] {
+ assert_eq!(batch.num_rows(), target_batch_size);
+ }
+
+ let actual = concat_batches(&schema, completed.iter()).unwrap();
+ let expected = take_record_batch(&batch, &indices).unwrap();
+ assert_eq!(expected, actual);
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_empty_is_noop() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..4i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from(Vec::<u32>::new());
+
+ let mut coalescer = BatchCoalescer::new(schema, 4096);
+ coalescer.push_batch_with_indices(batch, &indices).unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ assert!(coalescer.next_completed_batch().is_none());
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_schema_mismatch() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Int32, false),
+ ]));
+ let mut coalescer = BatchCoalescer::new(schema, 100);
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)])),
+ vec![Arc::new(Int32Array::from_iter_values(0..5i32))],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1]);
+ let err = coalescer
+ .push_batch_with_indices(batch, &indices)
+ .unwrap_err()
+ .to_string();
+ assert!(
+ err.contains("Batch has 1 columns but BatchCoalescer expects 2"),
+ "unexpected error: {err}"
+ );
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_oob_before_mutation() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Utf8View, false),
+ ]));
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..16i32)),
+ Arc::new(StringViewArray::from(vec!["a"; 16])),
+ ],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1, 2, 3, 0, 1, 2, 99]);
Review Comment:
added in latest commit, `test_push_batch_with_indices_rejects_oob_in_direct_copy_chunk`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4318089463
##########
arrow-select/src/coalesce.rs:
##########
@@ -40,6 +41,45 @@ use fixed_size_binary::InProgressFixedSizeBinaryArray;
use generic::GenericInProgressArray;
use primitive::InProgressPrimitiveArray;
+// Caller must have bounds-checked `indices` against `source_nulls`.
+pub(crate) fn append_gathered_nulls(
+ nulls: &mut NullBufferBuilder,
+ source_nulls: &NullBuffer,
+ indices: &[u32],
+) {
+ if source_nulls.null_count() == 0 {
+ nulls.append_n_non_nulls(indices.len());
+ return;
+ }
+ let buf = Buffer::from_slice_ref(indices);
+ let scalar: ScalarBuffer<u32> = ScalarBuffer::new(buf, 0, indices.len());
+ let indices_arr = UInt32Array::new(scalar, None);
+ let gathered = take_bits::<UInt32Type, false>(source_nulls.inner(), &indices_arr);
+ nulls.append_buffer(&NullBuffer::new(gathered));
+}
Review Comment:
For sparse selections with nulls, the direct path gains much less than without them. In both head-commit runs, `take: primitive, 8192, nulls: 0.1, selectivity: 0.001` takes about 33ms against about 4.9ms for `nulls: 0` (1.42x and 1.41x over base, against 6.20x and 6.29x).
Most of that time is spent in this function. I sampled that case on an Apple M5 Max, and `append_gathered_nulls` held 83% of the samples under `take_streams`. Within it, `Buffer::from_slice_ref` took 22% and dropping the temporary `UInt32Array` took 14%, so copying and freeing the indices costs more than the `take_bits` gather itself (16%).
What do you think about passing the caller's `UInt32Array` down instead, sliced per chunk in `push_batch_with_index`, and handing it to `take_bits` directly? [`PrimitiveArray::slice`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-array/src/array/primitive_array.rs#L865-L871) only clones the buffer's `Arc`, so the per-column copy, allocation, and free of the indices go away. I tried it against the head code in a single benchmark run, using a runtime switch between the two versions in the same process (times are criterion medians):
| case, `nulls: 0.1` | head | sliced `UInt32Array` |
| --- | --- | --- |
| `primitive`, 0.001 | 12.69ms | 7.73ms |
| `primitive`, 0.01 | 2.20ms | 1.57ms |
| `primitive`, 0.1 | 1.07ms | 0.96ms |
| `single_utf8view`, 0.001 | 6.29ms | 4.69ms |
| `single_utf8view`, 0.01 | 2.01ms | 1.77ms |
The `nulls: 0` cases don't reach this function, and they moved by less than 1% between the two. On aarch64 the head version makes four calls the sliced version doesn't:
```
; head
bl Buffer::from_slice_ref::<u32, &[u32]> ; allocate and copy the indices
bl ScalarBuffer::new
bl PrimitiveArray<UInt32Type>::try_new
bl take_bits::<UInt32Type, false>
bl NullBuffer::new
bl NullBufferBuilder::append_buffer
bl Arc<Bytes>::drop_slow ; free the gathered bits
bl drop_glue::<PrimitiveArray<..>> ; free the index copy
; sliced UInt32Array
bl take_bits::<UInt32Type, false>
bl NullBuffer::new
bl NullBufferBuilder::append_buffer
bl Arc<Bytes>::drop_slow
```
x86_64 has the same calls. The cost is a signature change to `InProgressArray::copy_rows_by_indices` so that it takes the `UInt32Array`. The trait is private, so the public API stays the same. The gain is smallest at higher selectivity, where the per-call cost is spread over more rows (10% at 0.1). I also tried a per-index `nulls.append(source_nulls.is_valid(idx as usize))` loop. It was faster still at 0.001 (3.93ms), but slower from 0.01 on (2.86ms at 0.01 and 2.85ms at 0.1), so it would need a length cutoff.
##########
arrow-select/src/coalesce.rs:
##########
@@ -335,11 +396,71 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ let exceeds_byte_view_threshold = self.has_byte_view_columns
+ && !should_use_byte_view_indexed_copy(batch.num_rows(), indices.len());
+ if exceeds_byte_view_threshold
+ && let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
What should `push_batch_with_indices` do with an out-of-bounds index? At the head commit, the answer depends on the schema and on how many indices there are. I pushed a 64-row batch with `n` indices, the last of which is `99`:
| column | `n` | result |
| --- | --- | --- |
| `Int32` | 3 | `Err(take index 99 out of bounds for array of 64 rows)` |
| `Int32` | 40 | panic: `index out of bounds: the len is 64 but the index is 99` |
| `Utf8View` | 3 | `Err(...)` |
| `Utf8View` | 40 | `Err(...)` (from this block) |
The panic comes from [`take_native`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/take.rs#L604-L608), which [`take_record_batch`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/take.rs#L1716-L1722) uses on the first column. Before this PR every case panicked there. Should we validate once at the top for all non-null `UInt32` indices, so every path returns the same error, and say so in the doc comment? That would also replace this byte-view-only block.
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let buf = Buffer::from_slice_ref(indices);
+ let scalar: ScalarBuffer<u32> = ScalarBuffer::new(buf, 0, indices.len());
+ let indices_arr = UInt32Array::new(scalar, None);
+ let taken = take(source.as_ref(), &indices_arr, None)?;
+ self.set_source(Some(taken));
+ let result = self.copy_rows(0, indices.len());
+ self.set_source(None);
Review Comment:
As I read `push_batch_with_indices`, this default never runs. The direct path requires [`!self.has_non_specialized_filter_columns`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/coalesce.rs#L415-L418), so every column is a primitive or view type, and both `InProgressPrimitiveArray` and `InProgressByteViewArray` override this method. Should we remove it, along with the `take` import that only it uses? The other option is to relax that gate so generic columns use this default while the primitive and view columns of a mixed schema still get the direct copy. That changes which paths the benchmarks exercise, so it would need its own numbers.
##########
arrow-select/src/coalesce.rs:
##########
@@ -2893,4 +3048,147 @@ mod tests {
"draining must release the accounted memory"
);
}
+
+ #[test]
+ fn test_push_batch_with_indices_splits_at_target_batch_size() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..1000i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from_iter_values((0..1000u32).step_by(3));
+ let target_batch_size = 100;
+
+ let mut coalescer = BatchCoalescer::new(Arc::clone(&schema), target_batch_size);
+ coalescer
+ .push_batch_with_indices(batch.clone(), &indices)
+ .unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ let mut completed = vec![];
+ while let Some(batch) = coalescer.next_completed_batch() {
+ completed.push(batch);
+ }
+ for batch in &completed[..completed.len() - 1] {
+ assert_eq!(batch.num_rows(), target_batch_size);
+ }
+
+ let actual = concat_batches(&schema, completed.iter()).unwrap();
+ let expected = take_record_batch(&batch, &indices).unwrap();
+ assert_eq!(expected, actual);
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_empty_is_noop() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..4i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from(Vec::<u32>::new());
+
+ let mut coalescer = BatchCoalescer::new(schema, 4096);
+ coalescer.push_batch_with_indices(batch, &indices).unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ assert!(coalescer.next_completed_batch().is_none());
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_schema_mismatch() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Int32, false),
+ ]));
+ let mut coalescer = BatchCoalescer::new(schema, 100);
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)])),
+ vec![Arc::new(Int32Array::from_iter_values(0..5i32))],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1]);
+ let err = coalescer
+ .push_batch_with_indices(batch, &indices)
+ .unwrap_err()
+ .to_string();
+ assert!(
+ err.contains("Batch has 1 columns but BatchCoalescer expects 2"),
+ "unexpected error: {err}"
+ );
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_oob_before_mutation() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Utf8View, false),
+ ]));
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..16i32)),
+ Arc::new(StringViewArray::from(vec!["a"; 16])),
+ ],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1, 2, 3, 0, 1, 2, 99]);
Review Comment:
As I read the thresholds, this test never reaches `push_batch_with_index`. With 8 indices over 16 rows and a `Utf8View` column, `should_use_byte_view_indexed_copy(16, 8)` is false, so the error comes from the validation block at the top of `push_batch_with_indices`, ahead of the `take_record_batch` fallback. Could we add a case that goes through the direct copy? One example is a 64-row batch with indices `[0, 1, 99]` and a `target_batch_size` of 2, so the bad index falls in the second chunk. That's the path where a partial write could happen.
##########
arrow-select/src/coalesce/byte_view.rs:
##########
@@ -354,13 +362,157 @@ impl<B: ByteViewType> InProgressByteViewArray<B> {
self.views.extend(new_views);
self.current = Some(dst_buffer);
}
+
+ #[inline(never)]
+ fn append_indexed_views_and_copy_strings(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ ) {
+ self.append_indexed_views_and_copy_strings_with_max_buffer_size(
+ src_views,
+ indices,
+ src_buffers,
+ selected_bytes,
+ i32::MAX as usize,
+ );
+ }
+
+ fn append_indexed_views_and_copy_strings_with_max_buffer_size(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ max_buffer_size: usize,
+ ) {
+ if selected_bytes == 0 {
+ self.views.reserve(indices.len());
+ for &idx in indices {
+ // SAFETY: caller validated idx < src_views.len().
+ let raw = unsafe { *src_views.get_unchecked(idx as usize) };
+ self.views.push(raw);
+ }
+ return;
+ }
+
+ let mut dst_buffer = match self.current.take() {
+ Some(current)
+ if current.capacity() - current.len() >= selected_bytes
+ && current
+ .len()
+ .checked_add(selected_bytes)
+ .is_some_and(|len| len <= max_buffer_size) =>
+ {
+ current
+ }
+ Some(current) => {
+ let completed_buffer: Buffer = current.into();
+ self.completed_buffers_size += completed_buffer.capacity();
+ self.completed.push(completed_buffer);
+ self.buffer_source
+ .next_buffer(selected_bytes.min(max_buffer_size))
+ }
+ None => self
+ .buffer_source
+ .next_buffer(selected_bytes.min(max_buffer_size)),
+ };
+
+ let current_len = self.views.len();
+ self.views.reserve(indices.len());
+ let output = self.views.spare_capacity_mut().as_mut_ptr().cast::<u128>();
+ let mut copied_bytes = 0usize;
+ for (i, &idx) in indices.iter().enumerate() {
+ // SAFETY: caller validated idx < src_views.len().
+ let raw = unsafe { *src_views.get_unchecked(idx as usize) };
+ let mut view = ByteView::from(raw);
+ if view.length > MAX_INLINE_VIEW_LEN {
+ let buffer_index = view.buffer_index as usize;
+ let buffer_offset = view.offset as usize;
+ let str_len = view.length as usize;
+ assert!(str_len <= max_buffer_size);
+
+ let dst_end = dst_buffer
+ .len()
+ .checked_add(str_len)
+ .expect("byte view buffer size overflow");
+ if dst_end > max_buffer_size {
+ let completed_buffer: Buffer = dst_buffer.into();
+ self.completed_buffers_size += completed_buffer.capacity();
+ self.completed.push(completed_buffer);
+
+ let remaining = selected_bytes
+ .checked_sub(copied_bytes)
+ .expect("selected byte count mismatch");
+ dst_buffer = self
+ .buffer_source
+ .next_buffer(remaining.min(max_buffer_size));
+ }
+
+ // SAFETY: source views carry valid buffer_index/offset/length.
+ let src = unsafe {
+ src_buffers
+ .get_unchecked(buffer_index)
+ .get_unchecked(buffer_offset..buffer_offset + str_len)
+ };
+ view.offset = dst_buffer
+ .len()
+ .try_into()
+ .expect("byte view offset exceeds u32::MAX");
+ view.buffer_index = self.completed.len().try_into().expect("too many buffers");
+ let dst_len = dst_buffer.len();
+ // SAFETY: the destination was sized for the remaining selected bytes,
+ // and `src` belongs to the input array and cannot overlap it.
+ unsafe {
+ std::ptr::copy_nonoverlapping(
+ src.as_ptr(),
+ dst_buffer.as_mut_ptr().add(dst_len),
+ str_len,
+ );
+ dst_buffer.set_len(dst_len + str_len);
+ }
Review Comment:
The neighboring `append_views_and_copy_strings_inner` copies with [`dst_buffer.extend_from_slice(src)`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/coalesce/byte_view.rs#L350-L357). This loop uses `copy_nonoverlapping` plus `set_len`, and its safety depends on the capacity reasoning across the reuse, new-buffer, and split branches above. Could we use `extend_from_slice` here too? It keeps the copy safe at the cost of one capacity compare per non-inline string. If a benchmark shows that compare matters, a `debug_assert!(dst_buffer.capacity() - dst_len >= str_len)` like the [capacity check in the neighbor](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/coalesce/byte_view.rs#L316-L337) would at least check the reasoning in tests.
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
Review Comment:
`copy_rows_by_indices` is a safe method, but the primitive and byte-view impls read through `indices` with unchecked pointer arithmetic. A caller that skips `validate_indices_bounds` gets undefined behavior rather than an error. `append_gathered_nulls` has the same precondition, because it calls `take_bits::<_, false>`. Should these be `unsafe fn` with a `# Safety` section, the way [`FilterBuilder::with_count`](https://github.com/apache/arrow-rs/blob/41bbe160f8e526fb621f0e2de3ebdead08cf45d2/arrow-select/src/filter.rs#L301) states its precondition? A `debug_assert!` on the maximum index in each impl would also catch a missed check in tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6275102004
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274138911)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (41bbe160f8e526fb621f0e2de3ebdead08cf45d2) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..41bbe160f8e526fb621f0e2de3ebdead08cf45d2)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.37 13.7±0.10ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.49 3.3±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.02ms ? ?/sec 1.01 2.8±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.36 20.5±0.05ms ? ?/sec 1.00 15.0±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.11 4.5±0.01ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.1±0.02ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 851.8±1.43µs ? ?/sec 1.01 863.2±0.82µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.97 12.9±0.12ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.03 2.3±0.01ms ? ?/sec 1.00 1129.4±17.34µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1590.9±10.76µs ? ?/sec 1.00 1596.2±5.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1088.9±2.71µs ? ?/sec 1.01 1101.2±4.33µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.5±0.04ms ? ?/sec 1.00 14.4±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.13 3.4±0.01ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.01 1945.2±6.51µs ? ?/sec 1.00 1931.6±4.67µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 846.6±1.54µs ? ?/sec 1.01 856.7±0.70µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 108.3±0.90ms ? ?/sec 1.00 108.2±0.92ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.3±0.01ms ? ?/sec 1.01 5.3±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.02 3.7±0.02ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.01ms ? ?/sec 1.00 1989.8±12.37µs ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.02 107.2±0.52ms ? ?/sec 1.00 105.4±0.57ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.01 5.7±0.03ms ? ?/sec 1.00 5.6±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.5±0.01ms ? ?/sec 1.01 3.6±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 164.3±0.48ms ? ?/sec 1.00 164.4±0.53ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.01 22.8±0.05ms ? ?/sec 1.00 22.7±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.01 6.5±0.01ms ? ?/sec 1.00 6.4±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.01 5.9±0.08ms ? ?/sec 1.00 5.8±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 5.3±0.07ms ? ?/sec 1.00 5.3±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.5±0.13ms ? ?/sec 1.01 32.8±0.17ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.0±0.04ms ? ?/sec 1.00 8.0±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.6±0.05ms ? ?/sec 1.02 5.7±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.8±0.06ms ? ?/sec 1.00 4.7±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.43 13.5±0.05ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.57 3.2±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1984.0±19.96µs ? ?/sec 1.00 1992.7±12.57µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.37 20.2±0.03ms ? ?/sec 1.00 14.8±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.18 4.4±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.01 2.9±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 845.1±1.12µs ? ?/sec 1.01 856.9±0.84µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.05 12.7±0.04ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.05 2.2±0.01ms ? ?/sec 1.00 1093.4±12.48µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1297.3±2.98µs ? ?/sec 1.01 1306.0±3.72µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 559.4±0.48µs ? ?/sec 1.03 578.4±1.07µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.6±0.03ms ? ?/sec 1.00 14.2±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.17 3.4±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1666.7±2.42µs ? ?/sec 1.01 1678.6±5.75µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 846.1±1.13µs ? ?/sec 1.02 867.0±1.69µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.4±0.03ms ? ?/sec 1.06 12.1±0.08ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.20 30.2±0.43ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.45 4.2±0.07ms ? ?/sec 1.00 1726.4±32.01µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.39 1664.7±4.18µs ? ?/sec 1.00 1200.3±1.40µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.01 829.4±0.85µs ? ?/sec 1.00 817.3±0.75µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.42 47.4±0.12ms ? ?/sec 1.00 33.3±0.16ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.14 6.9±0.02ms ? ?/sec 1.00 6.0±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.11 2.6±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1505.6±2.28µs ? ?/sec 1.03 1544.0±1.33µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.7±0.10ms ? ?/sec 1.00 23.7±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.01ms ? ?/sec 1.01 5.2±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.02 2.0±0.01ms ? ?/sec 1.00 1997.8±1.79µs ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.06ms ? ?/sec 1.01 33.2±0.11ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.4±0.13ms ? ?/sec 1.02 6.6±0.16ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.01 3.0±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.19 13.5±0.22ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.46 3.0±0.01ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.01ms ? ?/sec 1.02 3.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 806.3±0.68µs ? ?/sec 1.06 850.8±1.18µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.36 20.1±0.09ms ? ?/sec 1.00 14.8±0.09ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.19 4.1±0.01ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.01 3.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1056.7±1.14µs ? ?/sec 1.03 1091.1±1.39µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.05ms ? ?/sec 1.01 13.7±0.07ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1566.6±9.12µs ? ?/sec 1.01 1575.0±7.80µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 358.6±0.49µs ? ?/sec 1.01 361.7±1.01µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.01 241.8±0.55µs ? ?/sec 1.00 240.5±0.28µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.4±0.07ms ? ?/sec 1.01 20.5±0.13ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 619.1±1.00µs ? ?/sec 1.00 621.8±1.11µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.0±0.28µs ? ?/sec 1.01 434.8±0.39µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.3±0.03ms ? ?/sec 1.04 12.8±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1704.6±8.65µs ? ?/sec 1.03 1753.1±8.52µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 691.7±1.34µs ? ?/sec 1.03 715.6±1.75µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.02 459.0±0.70µs ? ?/sec 1.00 448.1±0.83µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.2±0.05ms ? ?/sec 1.02 19.7±0.14ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.02 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1016.0±0.95µs ? ?/sec 1.02 1039.3±2.00µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 697.9±1.21µs ? ?/sec 1.00 695.1±1.00µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.1±0.11ms ? ?/sec 1.03 13.6±0.12ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.01 1941.2±2.95µs ? ?/sec 1.00 1914.9±5.49µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1440.7±5.16µs ? ?/sec 1.00 1445.8±4.63µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.01 21.0±0.11ms ? ?/sec 1.00 20.8±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.6±0.01ms ? ?/sec 1.05 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.2±0.00ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1673.0±3.70µs ? ?/sec 1.00 1674.5±3.93µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.5±0.29ms ? ?/sec 1.03 35.4±0.11ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.01 5.1±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1876.1±4.52µs ? ?/sec 1.04 1943.2±4.05µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1187.4±1.53µs ? ?/sec 1.01 1198.9±2.10µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.3±0.11ms ? ?/sec 1.02 59.2±0.22ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.03ms ? ?/sec 1.01 11.0±0.04ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.3±0.00ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.00ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.0±0.13ms ? ?/sec 1.01 44.5±0.21ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.6±0.02ms ? ?/sec 1.00 9.6±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.0±0.01ms ? ?/sec 1.01 5.1±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.6±0.02ms ? ?/sec 1.01 3.7±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.5±0.17ms ? ?/sec 1.01 51.8±0.21ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.03 10.8±0.19ms ? ?/sec 1.00 10.5±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.02 5.9±0.02ms ? ?/sec 1.00 5.8±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.03ms ? ?/sec 1.01 4.3±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.0±0.15ms ? ?/sec 1.00 36.0±0.12ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.3±0.03ms ? ?/sec 1.00 12.3±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.01 12.7±0.15ms ? ?/sec 1.00 12.6±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.01 12.8±0.18ms ? ?/sec 1.00 12.7±0.16ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.2±0.16ms ? ?/sec 1.01 46.6±0.15ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.01 13.7±0.05ms ? ?/sec 1.00 13.6±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 11.5±0.14ms ? ?/sec 1.01 11.6±0.17ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 10.9±0.14ms ? ?/sec 1.00 10.8±0.15ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.17 13.4±0.19ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.45 2.9±0.01ms ? ?/sec 1.00 1988.4±19.75µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.5±0.00ms ? ?/sec 1.04 2.6±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 810.1±0.70µs ? ?/sec 1.05 850.8±1.27µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.42 20.2±0.06ms ? ?/sec 1.00 14.2±0.07ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.20 4.0±0.01ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.04 2.8±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1052.5±3.56µs ? ?/sec 1.03 1080.9±1.61µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-14meshs7-78b152e9-zzvz
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6274138911-3605-9d9br 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.4 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1366.9s |
| CPU sys | 21.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 59.9 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1381.4s |
| CPU sys | 21.3s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6274138911:finish:e19840b41a4ebb7d09a53092b9f5ddeb -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4329882347
##########
arrow-select/src/coalesce.rs:
##########
@@ -336,11 +404,81 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ if let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
> Also this seems like it validates the indices, and then calls take_record_batch which validates the indices again
this is true, we could use https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/take.rs#L177 instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6300584970
thank you for the review
> I think we need to make the relevant API as unsafe
> If we are going to make an unsafe take API that might be worth doing too
@alamb
is there any preference as to which direction we go in?
1. add validation and keep the unsafe code blocks in safe code
2. make the functions unsafe and push the validation onto the caller
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6303792376
> Specifically I think the API is `unsafe` in the sense that it could cause memory violations using only safe APIs and thus I don't think we can merge it as is.
Thanks @alamb. Sorry I didn't push harder on this. I asked for `unsafe fn` in the [first round](https://github.com/apache/arrow-rs/pull/11174#discussion_r4318098381), then accepted that the trait is private and that `copy_rows` already relies on caller-computed ranges, and I suggested `debug_assert!`s instead.
The precedent I saw was `InProgressArray::copy_rows`. It is a safe trait method with no documented precondition, and its impls call `get_unchecked` on `offset..offset + len` behind a `SAFETY` comment that relies on the caller ([primitive.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/primitive.rs#L167-L169), [byte_view.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/byte_view.rs#L411-L412)). Should I file an issue to make it `unsafe` or have it check the range?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4202741049
##########
arrow-select/src/coalesce/primitive.rs:
##########
@@ -206,6 +206,43 @@ impl<T: ArrowPrimitiveType + Debug> InProgressArray for InProgressPrimitiveArray
}
}
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
+ ) -> Result<(), ArrowError> {
Review Comment:
Validate every row index against batch.num_rows() before mutating any in-progress column, or make this API unsafe and document the precondition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241929860
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216906)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (40874968ff2a73c735b384cd46f3bb24f191e46c) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..40874968ff2a73c735b384cd46f3bb24f191e46c)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.59 13.8±0.19ms ? ?/sec 1.00 3.0±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.60 3.3±0.02ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.08 2.9±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1917.9±4.59µs ? ?/sec 1.19 2.3±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.39 20.7±0.21ms ? ?/sec 1.00 14.9±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.15 4.4±0.02ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.01ms ? ?/sec 1.08 3.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 856.4±1.32µs ? ?/sec 1.01 867.2±1.70µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.34 12.9±0.20ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.04 2.3±0.01ms ? ?/sec 1.00 1130.6±11.79µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.04 1604.0±3.76µs ? ?/sec 1.00 1545.7±7.13µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1093.5±3.99µs ? ?/sec 1.01 1109.6±3.47µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.42 19.9±0.21ms ? ?/sec 1.00 14.0±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.19 3.5±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1942.7±3.29µs ? ?/sec 1.03 2.0±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 855.3±0.86µs ? ?/sec 1.01 865.4±0.97µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 92.3±1.02ms ? ?/sec 1.03 95.1±0.97ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.3±0.01ms ? ?/sec 1.07 5.6±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.6±0.02ms ? ?/sec 1.08 3.9±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.02 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 89.3±0.80ms ? ?/sec 1.08 96.2±1.67ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.6±0.02ms ? ?/sec 1.04 5.8±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.6±0.02ms ? ?/sec 1.04 3.7±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.02 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.01 157.9±3.53ms ? ?/sec 1.00 157.0±2.95ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.4±0.09ms ? ?/sec 1.03 23.1±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 6.5±0.02ms ? ?/sec 1.02 6.6±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.04ms ? ?/sec 1.05 5.4±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.3±0.06ms ? ?/sec 1.06 4.6±0.10ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.7±0.20ms ? ?/sec 1.00 32.8±0.08ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.0±0.02ms ? ?/sec 1.02 8.2±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.0±0.04ms ? ?/sec 1.05 5.2±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.0±0.07ms ? ?/sec 1.01 4.0±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.68 13.6±0.11ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.59 3.3±0.01ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.06 2.5±0.01ms ? ?/sec 1.00 2.4±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1910.9±6.65µs ? ?/sec 1.03 1977.5±23.49µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.39 20.7±0.14ms ? ?/sec 1.00 14.9±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.19 4.4±0.02ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.02 2.8±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 847.9±0.97µs ? ?/sec 1.02 865.7±0.88µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.40 12.9±0.11ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.07 2.2±0.01ms ? ?/sec 1.00 1068.3±5.19µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.07 1331.8±6.75µs ? ?/sec 1.00 1246.2±3.95µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 570.8±0.80µs ? ?/sec 1.04 591.1±1.45µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.43 19.8±0.17ms ? ?/sec 1.00 13.8±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.19 3.5±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1709.0±4.11µs ? ?/sec 1.01 1734.4±2.39µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 853.5±1.45µs ? ?/sec 1.01 861.0±2.25µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.4±0.08ms ? ?/sec 1.03 11.7±0.16ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.27 30.4±0.28ms ? ?/sec 1.00 4.8±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.91 4.3±0.03ms ? ?/sec 1.00 1463.0±9.42µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.33 1646.5±3.26µs ? ?/sec 1.00 1240.2±2.09µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 821.6±1.26µs ? ?/sec 1.02 841.9±1.18µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.45 48.2±0.43ms ? ?/sec 1.00 33.4±0.11ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.14 7.0±0.06ms ? ?/sec 1.00 6.1±0.03ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.06 2.6±0.00ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1539.9±1.56µs ? ?/sec 1.00 1543.8±2.33µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.01 23.6±0.15ms ? ?/sec 1.00 23.5±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.01ms ? ?/sec 1.03 5.2±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.06 3.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.02 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.0±0.07ms ? ?/sec 1.00 33.0±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.3±0.04ms ? ?/sec 1.03 6.4±0.06ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.03ms ? ?/sec 1.08 3.1±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.1±0.03ms ? ?/sec 1.02 2.2±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.33 13.4±0.24ms ? ?/sec 1.00 4.0±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.61 2.9±0.01ms ? ?/sec 1.00 1811.1±4.45µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.00ms ? ?/sec 1.10 3.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 829.7±1.12µs ? ?/sec 1.03 857.8±1.00µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.43 20.5±0.22ms ? ?/sec 1.00 14.3±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.20 4.1±0.02ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.12 3.8±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1079.4±1.32µs ? ?/sec 1.02 1103.0±1.21µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.7±0.06ms ? ?/sec 1.01 13.8±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1572.5±9.31µs ? ?/sec 1.01 1592.1±10.12µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 355.5±0.98µs ? ?/sec 1.02 362.2±0.95µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 238.9±0.42µs ? ?/sec 1.01 242.3±0.24µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.01 20.4±0.07ms ? ?/sec 1.00 20.3±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 617.5±1.34µs ? ?/sec 1.01 623.5±1.44µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 432.0±0.23µs ? ?/sec 1.00 434.1±1.16µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.01 12.7±0.11ms ? ?/sec 1.00 12.6±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.04 1824.7±21.61µs ? ?/sec 1.00 1760.3±11.53µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 706.5±1.15µs ? ?/sec 1.06 748.7±3.49µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 439.9±0.76µs ? ?/sec 1.08 473.2±0.61µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.01 19.5±0.06ms ? ?/sec 1.00 19.3±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.02ms ? ?/sec 1.02 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 999.1±1.69µs ? ?/sec 1.09 1093.9±2.46µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 685.9±1.12µs ? ?/sec 1.03 707.2±1.29µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.01 13.5±0.10ms ? ?/sec 1.00 13.4±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 1986.2±2.01µs ? ?/sec 1.07 2.1±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1472.4±3.11µs ? ?/sec 1.06 1558.3±3.28µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.01 21.2±0.07ms ? ?/sec 1.00 21.0±0.08ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.02ms ? ?/sec 1.02 3.9±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.00ms ? ?/sec 1.04 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1704.2±2.45µs ? ?/sec 1.06 1798.7±12.57µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.8±0.23ms ? ?/sec 1.00 34.7±0.11ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.02 5.0±0.02ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1855.3±3.74µs ? ?/sec 1.07 1988.3±5.25µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1179.3±1.32µs ? ?/sec 1.05 1238.5±3.08µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.1±0.10ms ? ?/sec 1.01 58.7±0.14ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.6±0.01ms ? ?/sec 1.02 10.8±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.0±0.01ms ? ?/sec 1.04 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.01ms ? ?/sec 1.00 4.4±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.3±0.16ms ? ?/sec 1.00 44.4±0.12ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.1±0.12ms ? ?/sec 1.07 9.8±0.09ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.01ms ? ?/sec 1.05 5.4±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.03 3.8±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.0±0.22ms ? ?/sec 1.00 52.2±0.14ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.5±0.03ms ? ?/sec 1.05 11.0±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.02ms ? ?/sec 1.08 6.3±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.03ms ? ?/sec 1.03 4.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 35.9±0.18ms ? ?/sec 1.02 36.6±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.2±0.02ms ? ?/sec 1.03 12.6±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.2±0.09ms ? ?/sec 1.06 11.9±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.3±0.13ms ? ?/sec 1.19 12.3±0.46ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.8±0.18ms ? ?/sec 1.01 47.5±0.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.7±0.04ms ? ?/sec 1.06 14.5±0.05ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.1±0.07ms ? ?/sec 1.06 10.7±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.05 9.5±0.75ms ? ?/sec 1.00 9.1±0.12ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.42 13.3±0.19ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.69 2.9±0.01ms ? ?/sec 1.00 1703.6±2.37µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.00ms ? ?/sec 1.06 2.7±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 844.8±0.61µs ? ?/sec 1.02 863.6±1.58µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.45 20.2±0.19ms ? ?/sec 1.00 14.0±0.06ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.27 4.1±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.10 3.1±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1068.8±1.31µs ? ?/sec 1.02 1093.8±2.20µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-v23zn8nu-6db5fc75-j9wx
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6241216906-3554-lvc62 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 61.6 MiB |
| Avg memory | 20.9 MiB |
| CPU user | 1381.6s |
| CPU sys | 22.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.8 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 1369.3s |
| CPU sys | 23.4s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6241216906:finish:a256154baf6578dc1d671d46780c5542 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6275149642
I'll take a pass today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176518416
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6177484598
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176518416)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.78 13.6±0.13ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.77 3.5±0.07ms ? ?/sec 1.00 934.8±4.63µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 6.41 4.9±0.33ms ? ?/sec 1.00 768.3±2.23µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 7.71 3.6±0.33ms ? ?/sec 1.00 462.2±2.15µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.6±0.06ms ? ?/sec 1.00 15.2±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.82 4.7±0.05ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 3.27 4.1±0.29ms ? ?/sec 1.00 1261.9±4.59µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.11 886.7±4.38µs ? ?/sec 1.00 801.2±3.64µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.34 12.7±0.10ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.71 2.3±0.02ms ? ?/sec 1.00 867.4±6.47µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.15 1642.2±20.77µs ? ?/sec 1.00 762.5±1.74µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.45 1138.6±13.29µs ? ?/sec 1.00 465.2±1.80µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.36 19.6±0.05ms ? ?/sec 1.00 14.4±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.45 3.6±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.56 1974.1±16.37µs ? ?/sec 1.00 1266.5±11.55µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 856.5±6.12µs ? ?/sec 1.00 792.6±3.35µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 120.3±2.41ms ? ?/sec 1.01 121.2±1.67ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.08 6.1±0.27ms ? ?/sec 1.00 5.6±0.13ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.04 4.1±0.07ms ? ?/sec 1.00 3.9±0.10ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.02 2.2±0.04ms ? ?/sec 1.00 2.2±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.09 120.8±2.03ms ? ?/sec 1.00 110.6±2.70ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.05 6.2±0.13ms ? ?/sec 1.00 5.8±0.06ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.06 3.8±0.05ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.02 2.2±0.06ms ? ?/sec 1.00 2.1±0.03ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.06 193.5±3.83ms ? ?/sec 1.00 182.8±3.21ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.16 23.5±0.41ms ? ?/sec 1.00 20.3±0.46ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.05 7.1±0.10ms ? ?/sec 1.00 6.7±0.12ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.14 9.0±0.43ms ? ?/sec 1.00 7.9±0.36ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 7.7±0.25ms ? ?/sec 1.01 7.8±0.23ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.05 35.6±0.71ms ? ?/sec 1.00 33.8±0.43ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.02 8.5±0.14ms ? ?/sec 1.00 8.3±0.13ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.08 7.5±0.33ms ? ?/sec 1.00 6.9±0.44ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.05 6.6±0.25ms ? ?/sec 1.00 6.3±0.28ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.72 13.4±0.13ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.77 3.4±0.05ms ? ?/sec 1.00 910.5±6.11µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 6.06 4.7±0.36ms ? ?/sec 1.00 770.2±1.53µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 8.21 3.7±0.15ms ? ?/sec 1.00 453.3±1.88µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.36 20.6±0.10ms ? ?/sec 1.00 15.2±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.81 4.7±0.05ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.87 3.7±0.26ms ? ?/sec 1.00 1272.8±5.71µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.08 860.0±4.89µs ? ?/sec 1.00 798.0±3.95µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.69 12.8±0.10ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.75 2.3±0.03ms ? ?/sec 1.00 844.4±4.78µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.80 1369.8±12.75µs ? ?/sec 1.00 759.6±1.95µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.25 576.3±4.81µs ? ?/sec 1.00 461.6±1.27µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.8±0.17ms ? ?/sec 1.00 14.4±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.43 3.5±0.04ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.34 1701.4±8.28µs ? ?/sec 1.00 1266.5±4.59µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 858.8±4.23µs ? ?/sec 1.00 796.6±3.46µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.33 17.6±0.76ms ? ?/sec 1.00 13.3±1.12ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.19 30.3±0.24ms ? ?/sec 1.00 4.9±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.49 4.3±0.03ms ? ?/sec 1.00 1715.2±98.93µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.29 1678.0±14.35µs ? ?/sec 1.00 1304.3±2.44µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.01 834.0±2.63µs ? ?/sec 1.00 828.8±2.05µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.42 47.4±0.12ms ? ?/sec 1.00 33.5±0.15ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.13 7.0±0.03ms ? ?/sec 1.00 6.2±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.04 2.6±0.00ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1524.3±4.49µs ? ?/sec 1.09 1667.1±6.20µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.6±0.09ms ? ?/sec 1.22 28.7±0.15ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.11ms ? ?/sec 1.22 6.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.09 3.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.01ms ? ?/sec 1.10 2.3±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.2±0.10ms ? ?/sec 1.16 38.6±0.11ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.4±0.07ms ? ?/sec 1.21 7.8±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.11 3.3±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.11 2.4±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.81 13.7±0.21ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.90 3.0±0.01ms ? ?/sec 1.00 1039.7±6.15µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.60 3.2±0.02ms ? ?/sec 1.00 879.5±1.50µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.31 810.8±2.33µs ? ?/sec 1.00 619.4±0.90µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.30 19.9±0.13ms ? ?/sec 1.00 15.2±0.06ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.73 4.2±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.68 3.4±0.01ms ? ?/sec 1.00 1276.8±3.97µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.16 1054.9±1.57µs ? ?/sec 1.00 908.9±1.59µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.24ms ? ?/sec 1.32 17.9±0.07ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1571.7±22.98µs ? ?/sec 1.45 2.3±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 364.0±2.79µs ? ?/sec 1.71 621.2±1.33µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 242.7±0.38µs ? ?/sec 1.76 428.4±0.63µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.4±0.30ms ? ?/sec 1.25 25.4±0.13ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.03ms ? ?/sec 1.31 3.3±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 613.5±3.23µs ? ?/sec 1.46 896.7±1.20µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 432.1±0.63µs ? ?/sec 1.45 627.0±0.60µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 4.55 12.5±0.15ms ? ?/sec 1.00 2.7±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.85 1774.4±9.03µs ? ?/sec 1.00 959.8±1.82µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 696.7±2.14µs ? ?/sec 1.48 1031.6±1.95µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 459.0±2.50µs ? ?/sec 2.31 1059.9±19.17µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.50 19.2±0.04ms ? ?/sec 1.00 12.8±0.12ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.30 3.0±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1012.1±1.65µs ? ?/sec 1.41 1422.5±3.93µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 696.5±1.36µs ? ?/sec 1.62 1128.4±3.53µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 4.53 13.1±0.06ms ? ?/sec 1.00 2.9±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 2.16 2.5±0.01ms ? ?/sec 1.00 1136.5±6.09µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.37 2.0±0.02ms ? ?/sec 1.00 1463.2±7.32µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.32 1505.8±17.19µs ? ?/sec 1.00 1138.6±3.18µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.60 20.6±0.04ms ? ?/sec 1.00 12.9±0.08ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.45 3.7±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.25 2.3±0.01ms ? ?/sec 1.00 1852.9±6.84µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.19 1725.0±13.24µs ? ?/sec 1.00 1445.6±5.28µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 35.0±0.19ms ? ?/sec 1.14 40.0±0.17ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.02ms ? ?/sec 1.13 5.7±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1904.0±6.28µs ? ?/sec 1.15 2.2±0.00ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1217.7±7.79µs ? ?/sec 1.19 1446.6±14.11µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.6±0.17ms ? ?/sec 1.09 64.0±0.22ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.0±0.02ms ? ?/sec 1.03 11.3±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.3±0.01ms ? ?/sec 1.02 5.3±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.02ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.6±0.25ms ? ?/sec 1.11 49.6±0.16ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.2±0.09ms ? ?/sec 1.08 10.0±0.06ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.07ms ? ?/sec 1.05 5.5±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.9±0.06ms ? ?/sec 1.06 4.1±0.07ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.0±0.16ms ? ?/sec 1.09 56.9±0.17ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.04ms ? ?/sec 1.08 11.5±0.16ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.0±0.04ms ? ?/sec 1.06 6.3±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.5±0.05ms ? ?/sec 1.06 4.7±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.4±0.25ms ? ?/sec 1.27 46.2±0.53ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.8±0.20ms ? ?/sec 1.13 14.5±0.15ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 18.5±1.11ms ? ?/sec 1.04 19.2±0.91ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.01 18.1±0.92ms ? ?/sec 1.00 17.9±0.67ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 48.5±0.72ms ? ?/sec 1.15 55.6±0.53ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.5±0.25ms ? ?/sec 1.07 15.5±0.21ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 14.6±0.91ms ? ?/sec 1.00 14.6±0.79ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 15.2±0.78ms ? ?/sec 1.00 15.1±0.43ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.76 13.4±0.20ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.85 3.0±0.02ms ? ?/sec 1.00 1051.3±4.37µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 2.93 2.6±0.01ms ? ?/sec 1.00 898.2±2.21µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.31 814.2±2.35µs ? ?/sec 1.00 623.8±1.14µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.31 20.0±0.11ms ? ?/sec 1.00 15.3±0.11ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.70 4.1±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.11 2.8±0.01ms ? ?/sec 1.00 1310.8±4.36µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.16 1052.7±2.73µs ? ?/sec 1.00 908.3±1.53µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1r5gco5q-ad96bf51-brfx
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6176518416-3405-rzr4f 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.3 MiB |
| Avg memory | 20.5 MiB |
| CPU user | 1366.1s |
| CPU sys | 24.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1435.3s |
| Peak memory | 60.2 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1407.4s |
| CPU sys | 23.4s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6176518416:finish:2498930b6a50aed34e8417724911772e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4325841303
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &UInt32Array,
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let taken = take(source.as_ref(), indices, None)?;
+ self.set_source(Some(taken));
+ let result = self.copy_rows(0, indices.len());
+ self.set_source(None);
+ result
Review Comment:
yeah sounds like a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292061090
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292059412)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (3946b865b87fa51a8f28629b41fa266f0cf2b176) to 1357f25 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1357f25f26ca67634e992bdb35220b4ee59c780f..3946b865b87fa51a8f28629b41fa266f0cf2b176)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6292059412:start:7cac0ed984d56ff6250f42c5da07b750 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid closed pull request #11174: feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods
URL: https://github.com/apache/arrow-rs/pull/11174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6220020567
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219044770)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a002f6344e0ec90742c2870d74c5aeed792cca9f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a002f6344e0ec90742c2870d74c5aeed792cca9f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.34 13.5±0.08ms ? ?/sec 1.00 3.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.73 3.4±0.01ms ? ?/sec 1.00 1983.8±3.72µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.04 3.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.13 2.3±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.33 20.5±0.03ms ? ?/sec 1.00 15.4±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.22 4.6±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.2±0.03ms ? ?/sec 1.06 3.4±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 879.7±0.94µs ? ?/sec 1.59 1400.9±2.62µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.10 12.7±0.10ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 1.92 2.3±0.01ms ? ?/sec 1.00 1221.2±3.43µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.02 1699.1±2.91µs ? ?/sec 1.00 1667.1±6.55µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1130.3±3.75µs ? ?/sec 1.12 1263.7±12.25µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.7±0.03ms ? ?/sec 1.00 14.3±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.15 3.6±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.1±0.00ms ? ?/sec 1.03 2.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 872.2±0.75µs ? ?/sec 1.24 1085.0±1.93µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.14 113.1±4.76ms ? ?/sec 1.00 98.8±0.93ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.04 5.7±0.01ms ? ?/sec 1.00 5.5±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.01 3.8±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.01 98.2±0.93ms ? ?/sec 1.00 97.5±1.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.03 6.0±0.02ms ? ?/sec 1.00 5.9±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.02ms ? ?/sec 1.02 3.8±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.1±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.06 159.2±3.29ms ? ?/sec 1.00 150.8±2.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.01 22.9±0.06ms ? ?/sec 1.00 22.7±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.05 6.8±0.01ms ? ?/sec 1.00 6.5±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.03 5.4±0.04ms ? ?/sec 1.00 5.2±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.02 4.4±0.07ms ? ?/sec 1.00 4.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.02 32.8±0.11ms ? ?/sec 1.00 32.3±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.04 8.5±0.03ms ? ?/sec 1.00 8.2±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.04 5.3±0.03ms ? ?/sec 1.00 5.1±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.1±0.05ms ? ?/sec 1.00 4.0±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.37 13.3±0.06ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.74 3.4±0.02ms ? ?/sec 1.00 1940.5±3.31µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.04 2.7±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1981.0±6.65µs ? ?/sec 1.13 2.2±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.38 20.6±0.04ms ? ?/sec 1.00 14.9±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.24 4.5±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 867.1±1.26µs ? ?/sec 1.60 1390.3±3.39µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.19 12.7±0.06ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.11 2.4±0.02ms ? ?/sec 1.00 1126.4±2.93µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.03 1414.9±2.12µs ? ?/sec 1.00 1369.4±1.52µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 596.2±0.67µs ? ?/sec 1.21 721.4±1.49µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.7±0.04ms ? ?/sec 1.00 14.3±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.19 3.5±0.01ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1787.2±2.20µs ? ?/sec 1.03 1839.1±2.50µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 877.2±1.37µs ? ?/sec 1.34 1173.4±2.64µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.02 11.6±0.04ms ? ?/sec 1.00 11.3±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.24 30.3±0.30ms ? ?/sec 1.00 4.9±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.56 4.8±0.08ms ? ?/sec 1.00 1858.9±7.96µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.32 1767.5±5.34µs ? ?/sec 1.00 1344.1±5.58µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.02 849.2±4.03µs ? ?/sec 1.00 832.2±1.56µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.46 47.6±0.24ms ? ?/sec 1.00 32.7±0.15ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.12 7.0±0.03ms ? ?/sec 1.00 6.3±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.06 2.7±0.00ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1558.9±1.35µs ? ?/sec 1.07 1672.6±2.31µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.7±0.10ms ? ?/sec 1.00 23.6±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.05 3.0±0.00ms ? ?/sec 1.00 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.02 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.0±0.09ms ? ?/sec 1.01 33.5±0.23ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.4±0.04ms ? ?/sec 1.07 6.8±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.02 3.1±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.17 13.4±0.13ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.36 3.0±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.3±0.01ms ? ?/sec 1.13 3.8±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 825.7±0.78µs ? ?/sec 1.96 1620.4±5.07µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.35 20.0±0.07ms ? ?/sec 1.00 14.8±0.09ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.12 4.2±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.6±0.01ms ? ?/sec 1.17 4.2±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1079.3±1.71µs ? ?/sec 2.44 2.6±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.5±0.10ms ? ?/sec 1.01 13.6±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1556.1±3.51µs ? ?/sec 1.01 1564.7±9.56µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 361.7±0.96µs ? ?/sec 1.00 361.5±1.47µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 241.2±0.20µs ? ?/sec 1.00 241.3±0.49µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.2±0.10ms ? ?/sec 1.01 20.5±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 619.2±1.06µs ? ?/sec 1.01 625.8±1.27µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 430.7±0.38µs ? ?/sec 1.00 432.8±0.41µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.06ms ? ?/sec 1.00 12.6±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1687.6±7.74µs ? ?/sec 1.18 1992.7±10.77µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.02 723.2±0.89µs ? ?/sec 1.00 708.8±0.98µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.03 477.6±0.64µs ? ?/sec 1.00 465.9±0.57µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.2±0.03ms ? ?/sec 1.00 19.2±0.10ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.07 3.2±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1042.8±1.46µs ? ?/sec 1.02 1059.9±1.45µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.04 727.8±1.41µs ? ?/sec 1.00 701.0±0.95µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.02 13.4±0.32ms ? ?/sec 1.00 13.2±0.10ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.05 2.2±0.00ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.04 1581.3±1.83µs ? ?/sec 1.00 1526.6±4.23µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.5±0.08ms ? ?/sec 1.00 20.6±0.11ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.01 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.04 2.5±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.03 1814.1±2.75µs ? ?/sec 1.00 1760.0±4.87µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.8±0.10ms ? ?/sec 1.02 35.3±0.44ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.01 5.0±0.05ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1842.6±3.52µs ? ?/sec 1.03 1905.8±4.24µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.02 1251.5±2.03µs ? ?/sec 1.00 1225.5±7.28µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.3±0.14ms ? ?/sec 1.02 59.5±0.55ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.01ms ? ?/sec 1.01 11.0±0.06ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.3±0.01ms ? ?/sec 1.00 5.3±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.01 45.0±0.20ms ? ?/sec 1.00 44.5±0.14ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.8±0.10ms ? ?/sec 1.00 9.8±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.02 5.4±0.02ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.02 3.8±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.3±0.19ms ? ?/sec 1.00 52.2±0.09ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.01 10.9±0.04ms ? ?/sec 1.00 10.8±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.03 6.3±0.03ms ? ?/sec 1.00 6.1±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.5±0.03ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.8±0.20ms ? ?/sec 1.00 36.8±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.04 13.1±0.03ms ? ?/sec 1.00 12.6±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.04 11.9±0.13ms ? ?/sec 1.00 11.5±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.03 10.6±0.21ms ? ?/sec 1.00 10.3±0.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 47.5±0.18ms ? ?/sec 1.00 47.5±0.23ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.04 14.7±0.04ms ? ?/sec 1.00 14.1±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.04 10.9±0.14ms ? ?/sec 1.00 10.5±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 9.0±0.09ms ? ?/sec 1.00 8.9±0.12ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.25 13.1±0.11ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.52 3.0±0.00ms ? ?/sec 1.00 1945.3±5.01µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.10 3.1±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 819.6±1.43µs ? ?/sec 2.02 1652.5±3.14µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.42 20.0±0.05ms ? ?/sec 1.00 14.2±0.06ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.23 4.2±0.01ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.00ms ? ?/sec 1.17 3.5±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1071.9±1.02µs ? ?/sec 2.40 2.6±0.00ms ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1n7wuryf-44038fb1-q2m4
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6219044770-3511-4gll6 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 59.9 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1376.5s |
| CPU sys | 21.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1390.3s |
| Peak memory | 61.3 MiB |
| Avg memory | 20.6 MiB |
| CPU user | 1369.2s |
| CPU sys | 19.6s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6219044770:finish:6e5ed72758ffc763dce777fe58f4a106 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179340795
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179339186)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (4850db63ea2da3304d698c9a905c47fcb9c368a0) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4850db63ea2da3304d698c9a905c47fcb9c368a0)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6179339186:start:a43024c707c48f7835313073da818567 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179339186
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219045437
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219043135)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a002f6344e0ec90742c2870d74c5aeed792cca9f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a002f6344e0ec90742c2870d74c5aeed792cca9f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6219043135:start:b1c19f8d5be1735ac8cb15bf5522cb78 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219043135
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219044770
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6252933176
@alamb, @mbutrovich this is ready for review, I cant mark it as ready for review since I have 3 other PR's open 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6270439270
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6270102929)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (1818602ad10103a544d89c1decba5e1424329179) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..1818602ad10103a544d89c1decba5e1424329179)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (primitive|.*utf8view)'
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.53 13.7±0.08ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.63 3.3±0.04ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 3.0±0.06ms ? ?/sec 1.12 3.4±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.2±0.07ms ? ?/sec 1.02 2.2±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.37 20.3±0.11ms ? ?/sec 1.00 14.9±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.19 4.4±0.04ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.2±0.07ms ? ?/sec 1.13 3.6±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 856.0±1.50µs ? ?/sec 1.00 856.2±2.36µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.21 12.9±0.10ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.11 2.3±0.02ms ? ?/sec 1.00 1070.3±13.17µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.02 1345.6±7.75µs ? ?/sec 1.00 1314.9±6.37µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 572.0±1.26µs ? ?/sec 1.02 580.9±1.12µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.6±0.09ms ? ?/sec 1.00 14.3±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.23 3.4±0.02ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1718.9±5.11µs ? ?/sec 1.04 1787.3±3.88µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 857.4±1.90µs ? ?/sec 1.01 862.1±2.50µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.01 12.0±0.15ms ? ?/sec 1.00 11.9±0.13ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.38 30.1±0.35ms ? ?/sec 1.00 4.7±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.02 4.4±0.11ms ? ?/sec 1.00 1443.1±4.57µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.37 1680.8±11.56µs ? ?/sec 1.00 1230.4±2.15µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 826.1±4.29µs ? ?/sec 1.00 826.5±0.91µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.48 47.1±0.27ms ? ?/sec 1.00 31.8±0.15ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.21 7.0±0.18ms ? ?/sec 1.00 5.8±0.11ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.08 2.6±0.00ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1528.1±1.82µs ? ?/sec 1.00 1533.0±9.27µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.33 13.3±0.15ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.53 3.0±0.02ms ? ?/sec 1.00 1980.5±20.36µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.02ms ? ?/sec 1.18 3.1±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 809.0±3.35µs ? ?/sec 1.02 826.6±0.97µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.47 20.1±0.16ms ? ?/sec 1.00 13.7±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.22 4.1±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.19 3.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1061.2±3.34µs ? ?/sec 1.02 1077.4±5.56µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-185lwdpq-677e7f7e-8f4n
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6270102929-3593-2c64f 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 355.1s |
| Peak memory | 24.0 MiB |
| Avg memory | 15.6 MiB |
| CPU user | 350.8s |
| CPU sys | 0.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 350.1s |
| Peak memory | 23.2 MiB |
| Avg memory | 15.6 MiB |
| CPU user | 346.2s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6270102929:finish:803f5b17bb677f5d0c3df3f717c2885c -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321603142
##########
arrow-select/src/coalesce.rs:
##########
@@ -40,6 +41,45 @@ use fixed_size_binary::InProgressFixedSizeBinaryArray;
use generic::GenericInProgressArray;
use primitive::InProgressPrimitiveArray;
+// Caller must have bounds-checked `indices` against `source_nulls`.
+pub(crate) fn append_gathered_nulls(
+ nulls: &mut NullBufferBuilder,
+ source_nulls: &NullBuffer,
+ indices: &[u32],
+) {
+ if source_nulls.null_count() == 0 {
+ nulls.append_n_non_nulls(indices.len());
+ return;
+ }
+ let buf = Buffer::from_slice_ref(indices);
+ let scalar: ScalarBuffer<u32> = ScalarBuffer::new(buf, 0, indices.len());
+ let indices_arr = UInt32Array::new(scalar, None);
+ let gathered = take_bits::<UInt32Type, false>(source_nulls.inner(), &indices_arr);
+ nulls.append_buffer(&NullBuffer::new(gathered));
+}
Review Comment:
yea this makes a lot of sense to me. changing the API to `[u32]` or something of this nature allows for buffer re-use as well across calls which should be nice savings for medium/large sized buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274138911
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274351136
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225823879
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225097447)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.18 13.5±0.08ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.76 3.4±0.01ms ? ?/sec 1.00 1899.9±8.89µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.15ms ? ?/sec 1.00 3.1±0.12ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.12 2.4±0.08ms ? ?/sec 1.00 2.1±0.12ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.5±0.09ms ? ?/sec 1.00 15.2±0.08ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.25 4.6±0.03ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.22ms ? ?/sec 1.05 3.5±0.12ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.01 865.6±2.37µs ? ?/sec 1.00 854.1±3.77µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.79 12.6±0.10ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.02 2.3±0.01ms ? ?/sec 1.00 1126.0±2.71µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.04 1641.0±6.97µs ? ?/sec 1.00 1572.9±5.94µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.02 1113.9±7.87µs ? ?/sec 1.00 1094.0±4.90µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.5±0.04ms ? ?/sec 1.00 14.3±0.08ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.21 3.5±0.01ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.02 869.4±3.02µs ? ?/sec 1.00 854.5±2.61µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 111.1±4.70ms ? ?/sec 1.05 116.3±1.76ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.03 5.5±0.04ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.04 3.7±0.03ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.03 2.1±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.03 107.0±4.17ms ? ?/sec 1.00 103.6±8.85ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.05 5.9±0.04ms ? ?/sec 1.00 5.7±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.05 3.7±0.03ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.02 2.1±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.04 174.0±5.68ms ? ?/sec 1.00 167.9±4.37ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.6±0.07ms ? ?/sec 1.00 22.7±0.21ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.04 6.6±0.06ms ? ?/sec 1.00 6.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.08 6.6±0.33ms ? ?/sec 1.00 6.1±0.42ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.24 6.4±0.37ms ? ?/sec 1.00 5.1±0.62ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.03 33.0±0.19ms ? ?/sec 1.00 32.1±0.20ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.06 8.3±0.05ms ? ?/sec 1.00 7.9±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.01 6.2±0.31ms ? ?/sec 1.00 6.2±0.17ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.21 5.7±0.20ms ? ?/sec 1.00 4.7±0.56ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.26 13.3±0.07ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.79 3.3±0.01ms ? ?/sec 1.00 1851.7±5.17µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.09 2.9±0.13ms ? ?/sec 1.00 2.7±0.14ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.06 2.2±0.16ms ? ?/sec 1.00 2.1±0.12ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.3±0.04ms ? ?/sec 1.00 15.1±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.27 4.5±0.03ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.06 3.3±0.15ms ? ?/sec 1.00 3.1±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.03 863.6±2.44µs ? ?/sec 1.00 842.0±1.78µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.97 12.6±0.07ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.12 2.2±0.01ms ? ?/sec 1.00 1060.2±4.52µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.07 1347.5±3.65µs ? ?/sec 1.00 1254.0±2.61µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.04 585.4±1.48µs ? ?/sec 1.00 564.6±1.45µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.6±0.04ms ? ?/sec 1.00 14.2±0.09ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.26 3.5±0.01ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.01 1769.6±9.33µs ? ?/sec 1.00 1748.9±4.12µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.03 868.0±3.18µs ? ?/sec 1.00 845.6±1.58µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 12.1±0.53ms ? ?/sec 1.03 12.5±0.37ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 5.87 30.3±0.24ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.67 4.2±0.02ms ? ?/sec 1.00 1562.1±59.55µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.42 1678.1±2.49µs ? ?/sec 1.00 1183.8±1.36µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.04 846.8±2.06µs ? ?/sec 1.00 817.6±0.69µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.43 47.2±0.12ms ? ?/sec 1.00 33.0±0.20ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.16 6.9±0.02ms ? ?/sec 1.00 5.9±0.14ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.15 2.7±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.02 1543.3±1.97µs ? ?/sec 1.00 1509.2±1.61µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.5±0.10ms ? ?/sec 1.01 23.6±0.06ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.01 5.1±0.01ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.02 2.9±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.03 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.0±0.09ms ? ?/sec 1.00 32.9±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.01 6.4±0.06ms ? ?/sec 1.00 6.3±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.05 3.0±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.96 13.3±0.18ms ? ?/sec 1.00 4.5±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.43 3.0±0.02ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.01ms ? ?/sec 1.13 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.03 822.1±0.99µs ? ?/sec 1.00 800.4±1.23µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.33 20.2±0.14ms ? ?/sec 1.00 15.2±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.13 4.1±0.02ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.5±0.01ms ? ?/sec 1.13 4.0±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.04 1079.3±1.44µs ? ?/sec 1.00 1039.7±1.41µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.5±0.05ms ? ?/sec 1.03 13.9±0.22ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1552.9±4.95µs ? ?/sec 1.03 1604.5±24.22µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 363.0±0.60µs ? ?/sec 1.00 363.6±3.34µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.02 243.6±0.19µs ? ?/sec 1.00 239.5±0.29µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.3±0.04ms ? ?/sec 1.01 20.6±0.28ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.03ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 618.3±1.34µs ? ?/sec 1.02 631.0±3.42µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 433.8±0.29µs ? ?/sec 1.00 434.4±0.56µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.02ms ? ?/sec 1.01 12.6±0.08ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.02 1744.0±7.73µs ? ?/sec 1.00 1705.7±16.04µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.05 717.6±1.36µs ? ?/sec 1.00 681.9±1.48µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.09 486.1±0.61µs ? ?/sec 1.00 444.8±0.75µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.2±0.07ms ? ?/sec 1.01 19.3±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.02 3.1±0.00ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.05 1044.1±1.74µs ? ?/sec 1.00 996.5±1.15µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.02 701.4±0.93µs ? ?/sec 1.00 687.4±1.93µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.09ms ? ?/sec 1.01 13.2±0.04ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.3±0.01ms ? ?/sec 1.05 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.04 1986.4±5.35µs ? ?/sec 1.00 1909.4±4.39µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.07 1553.9±10.80µs ? ?/sec 1.00 1456.9±4.70µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.8±0.06ms ? ?/sec 1.00 20.9±0.06ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.01ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.09 2.4±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.06 1777.7±10.03µs ? ?/sec 1.00 1681.0±6.40µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.9±0.10ms ? ?/sec 1.00 35.0±0.21ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.01 5.1±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.04 1989.3±4.76µs ? ?/sec 1.00 1903.9±3.75µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.03 1231.7±1.82µs ? ?/sec 1.00 1199.8±7.81µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.1±0.10ms ? ?/sec 1.01 58.6±0.17ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.8±0.02ms ? ?/sec 1.01 10.9±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.02 5.3±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.5±0.18ms ? ?/sec 1.00 44.3±0.07ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.1±0.03ms ? ?/sec 1.00 9.0±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.06 5.3±0.02ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.03 3.7±0.02ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.0±0.12ms ? ?/sec 1.00 51.9±0.13ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.04ms ? ?/sec 1.00 10.7±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.05 6.2±0.03ms ? ?/sec 1.00 5.9±0.17ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.02 4.4±0.04ms ? ?/sec 1.00 4.4±0.16ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.1±0.17ms ? ?/sec 1.01 36.6±0.22ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.03 12.5±0.08ms ? ?/sec 1.00 12.2±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.09 13.9±0.66ms ? ?/sec 1.00 12.7±0.95ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.13 15.2±0.43ms ? ?/sec 1.00 13.4±0.90ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.03 47.5±0.22ms ? ?/sec 1.00 46.2±0.18ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.03 14.2±0.06ms ? ?/sec 1.00 13.8±0.09ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.05 12.5±0.91ms ? ?/sec 1.00 12.0±0.56ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.09 12.7±0.66ms ? ?/sec 1.00 11.6±0.89ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.97 13.2±0.17ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.57 2.9±0.01ms ? ?/sec 1.00 1836.1±5.86µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.00ms ? ?/sec 1.11 2.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 832.8±3.04µs ? ?/sec 1.00 834.3±14.13µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.36 19.9±0.09ms ? ?/sec 1.00 14.6±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.26 4.1±0.01ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.00ms ? ?/sec 1.11 3.2±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.05 1093.4±1.58µs ? ?/sec 1.00 1037.6±2.00µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-nwpa3evy-84ea4cfa-lv5z
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6225097447-3526-dkppg 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1380.3s |
| Peak memory | 60.6 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1353.5s |
| CPU sys | 21.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 59.2 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1378.5s |
| CPU sys | 22.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6225097447:finish:f37f9230d3d058e2ea050e9511c7d8a1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225097028
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225096087)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6225096087:start:060c532c9f2af96cb62a8665297d1d78 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225098938
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225097447)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6225097447:start:4eda232007c54c6a9a847d7c445803f3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6227337266
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226706836)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.29 13.8±0.11ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.60 3.3±0.01ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.02ms ? ?/sec 1.09 2.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1910.7±4.45µs ? ?/sec 1.01 1928.3±5.83µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.4±0.05ms ? ?/sec 1.00 15.2±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.18 4.4±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.13 3.3±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 855.2±1.24µs ? ?/sec 1.00 858.7±1.17µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.93 12.8±0.09ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.00 2.3±0.01ms ? ?/sec 1.00 1148.0±2.80µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1597.9±10.29µs ? ?/sec 1.01 1619.8±7.10µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1092.1±3.40µs ? ?/sec 1.00 1095.3±4.28µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.6±0.04ms ? ?/sec 1.00 14.3±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.20 3.5±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1941.7±3.87µs ? ?/sec 1.07 2.1±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 850.2±1.04µs ? ?/sec 1.00 854.3±0.62µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.03 99.4±1.20ms ? ?/sec 1.00 96.9±1.69ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.3±0.03ms ? ?/sec 1.02 5.4±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.7±0.02ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.02 97.5±1.20ms ? ?/sec 1.00 95.3±1.39ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.01 5.7±0.01ms ? ?/sec 1.00 5.7±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.6±0.02ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 151.8±2.38ms ? ?/sec 1.00 151.8±2.46ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.5±0.07ms ? ?/sec 1.00 22.5±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 6.4±0.04ms ? ?/sec 1.04 6.6±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.07ms ? ?/sec 1.01 5.1±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.3±0.07ms ? ?/sec 1.01 4.4±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.01 32.3±0.12ms ? ?/sec 1.00 32.1±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 7.9±0.02ms ? ?/sec 1.01 8.0±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.1±0.06ms ? ?/sec 1.00 5.1±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.0±0.10ms ? ?/sec 1.00 4.0±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.33 13.6±0.07ms ? ?/sec 1.00 3.1±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.68 3.2±0.01ms ? ?/sec 1.00 1926.5±11.28µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.5±0.01ms ? ?/sec 1.02 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1880.1±4.77µs ? ?/sec 1.01 1898.4±4.08µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.3±0.05ms ? ?/sec 1.00 15.0±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.21 4.3±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.09 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 850.1±0.81µs ? ?/sec 1.00 850.0±1.03µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.07 12.9±0.06ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.00 2.3±0.01ms ? ?/sec 1.00 1142.5±17.16µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1305.5±1.95µs ? ?/sec 1.00 1301.6±2.14µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 575.5±0.73µs ? ?/sec 1.01 581.8±1.16µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.7±0.06ms ? ?/sec 1.00 14.3±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.19 3.4±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1694.6±2.23µs ? ?/sec 1.07 1805.7±2.38µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 849.6±2.22µs ? ?/sec 1.00 851.5±0.60µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.03 12.1±0.10ms ? ?/sec 1.00 11.8±0.33ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 5.92 30.3±0.18ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.85 4.2±0.02ms ? ?/sec 1.00 1486.6±28.13µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.37 1669.8±3.10µs ? ?/sec 1.00 1218.5±2.51µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.01 832.5±0.77µs ? ?/sec 1.00 824.8±1.05µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.3±0.13ms ? ?/sec 1.00 33.6±0.11ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.14 6.9±0.03ms ? ?/sec 1.00 6.1±0.07ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.09 2.7±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1529.4±2.16µs ? ?/sec 1.00 1527.7±8.04µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.5±0.08ms ? ?/sec 1.00 23.5±0.12ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.02ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.9±0.01ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.09ms ? ?/sec 1.00 32.8±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.02 6.7±0.04ms ? ?/sec 1.00 6.6±0.17ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.03 13.4±0.13ms ? ?/sec 1.00 4.4±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.38 3.0±0.02ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.01ms ? ?/sec 1.18 3.7±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.02 837.9±1.40µs ? ?/sec 1.00 818.8±0.96µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.32 20.1±0.13ms ? ?/sec 1.00 15.2±0.04ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.13 4.0±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.19 4.0±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.03 1086.2±2.18µs ? ?/sec 1.00 1053.1±1.27µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.10ms ? ?/sec 1.01 13.7±0.03ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1574.2±10.96µs ? ?/sec 1.00 1578.1±5.41µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 360.0±0.81µs ? ?/sec 1.00 360.5±0.55µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.0±0.24µs ? ?/sec 1.00 240.1±0.74µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.3±0.10ms ? ?/sec 1.00 20.2±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 617.3±0.82µs ? ?/sec 1.00 616.3±1.26µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 433.2±0.26µs ? ?/sec 1.00 432.1±0.32µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.03 12.8±0.21ms ? ?/sec 1.00 12.5±0.03ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.01 1756.8±5.02µs ? ?/sec 1.00 1738.4±5.86µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.02 713.2±1.08µs ? ?/sec 1.00 699.8±1.30µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.04 468.5±0.90µs ? ?/sec 1.00 449.0±1.73µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.3±0.06ms ? ?/sec 1.00 19.3±0.03ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.01 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.01 1035.5±1.83µs ? ?/sec 1.00 1026.6±3.03µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 689.2±0.83µs ? ?/sec 1.01 696.9±1.45µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.02 13.3±0.16ms ? ?/sec 1.00 13.0±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 1955.5±2.47µs ? ?/sec 1.03 2.0±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1457.7±2.22µs ? ?/sec 1.01 1475.0±3.74µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.01 20.9±0.06ms ? ?/sec 1.00 20.6±0.03ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.01ms ? ?/sec 1.02 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.00ms ? ?/sec 1.01 2.3±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1697.3±1.93µs ? ?/sec 1.01 1719.6±5.14µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.7±0.27ms ? ?/sec 1.02 35.3±0.25ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.00 5.0±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.01 1909.1±4.12µs ? ?/sec 1.00 1898.7±5.12µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1201.5±2.21µs ? ?/sec 1.00 1206.7±3.19µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.3±0.08ms ? ?/sec 1.02 59.3±0.12ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.01ms ? ?/sec 1.00 10.7±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.0±0.00ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.01ms ? ?/sec 1.00 4.3±0.00ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.4±0.14ms ? ?/sec 1.00 44.3±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.02 9.6±0.03ms ? ?/sec 1.00 9.4±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.01ms ? ?/sec 1.01 5.1±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.2±0.12ms ? ?/sec 1.00 52.4±0.14ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.03 10.8±0.23ms ? ?/sec 1.00 10.5±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.02ms ? ?/sec 1.03 6.0±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.03ms ? ?/sec 1.00 4.3±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.0±0.23ms ? ?/sec 1.01 36.2±0.30ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.1±0.04ms ? ?/sec 1.05 12.7±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.3±0.15ms ? ?/sec 1.01 11.4±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.4±0.21ms ? ?/sec 1.01 10.5±0.20ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.7±0.27ms ? ?/sec 1.00 46.9±0.25ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.7±0.03ms ? ?/sec 1.02 13.9±0.05ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.3±0.16ms ? ?/sec 1.01 10.5±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 9.0±0.18ms ? ?/sec 1.00 9.0±0.20ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.02 13.2±0.10ms ? ?/sec 1.00 4.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.50 2.9±0.01ms ? ?/sec 1.00 1937.9±27.97µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.00ms ? ?/sec 1.16 3.0±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.03 844.9±0.70µs ? ?/sec 1.00 822.5±1.22µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.37 19.9±0.11ms ? ?/sec 1.00 14.6±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.24 4.1±0.01ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.19 3.3±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.01 1080.4±1.42µs ? ?/sec 1.00 1069.1±2.17µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-149lv8yb-5b0d3b2d-jhrf
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6226706836-3529-g882h 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1410.3s |
| Peak memory | 60.1 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 1386.8s |
| CPU sys | 19.7s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.2 MiB |
| Avg memory | 20.6 MiB |
| CPU user | 1369.5s |
| CPU sys | 21.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6226706836:finish:6fa2a93c37e15aa5fc42b19922228dd2 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176522437
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176520201)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6176520201:start:9de20e13c50b7aa8d0ac0d58d6561fae -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175113536
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274140283
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274138911)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (41bbe160f8e526fb621f0e2de3ebdead08cf45d2) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..41bbe160f8e526fb621f0e2de3ebdead08cf45d2)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6274138911:start:16ebe134f9ec6056f96bb9482888c321 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321627569
##########
arrow-select/src/coalesce.rs:
##########
@@ -335,11 +396,71 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ let exceeds_byte_view_threshold = self.has_byte_view_columns
+ && !should_use_byte_view_indexed_copy(batch.num_rows(), indices.len());
+ if exceeds_byte_view_threshold
+ && let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
I think we should consildate this into one error, being
```rust
index out of bounds: the len is 64 but the index is 99
```
avoiding panics is always good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321680067
##########
arrow-select/src/coalesce.rs:
##########
@@ -2893,4 +3048,147 @@ mod tests {
"draining must release the accounted memory"
);
}
+
+ #[test]
+ fn test_push_batch_with_indices_splits_at_target_batch_size() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..1000i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from_iter_values((0..1000u32).step_by(3));
+ let target_batch_size = 100;
+
+ let mut coalescer = BatchCoalescer::new(Arc::clone(&schema), target_batch_size);
+ coalescer
+ .push_batch_with_indices(batch.clone(), &indices)
+ .unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ let mut completed = vec![];
+ while let Some(batch) = coalescer.next_completed_batch() {
+ completed.push(batch);
+ }
+ for batch in &completed[..completed.len() - 1] {
+ assert_eq!(batch.num_rows(), target_batch_size);
+ }
+
+ let actual = concat_batches(&schema, completed.iter()).unwrap();
+ let expected = take_record_batch(&batch, &indices).unwrap();
+ assert_eq!(expected, actual);
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_empty_is_noop() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..4i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from(Vec::<u32>::new());
+
+ let mut coalescer = BatchCoalescer::new(schema, 4096);
+ coalescer.push_batch_with_indices(batch, &indices).unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ assert!(coalescer.next_completed_batch().is_none());
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_schema_mismatch() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Int32, false),
+ ]));
+ let mut coalescer = BatchCoalescer::new(schema, 100);
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)])),
+ vec![Arc::new(Int32Array::from_iter_values(0..5i32))],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1]);
+ let err = coalescer
+ .push_batch_with_indices(batch, &indices)
+ .unwrap_err()
+ .to_string();
+ assert!(
+ err.contains("Batch has 1 columns but BatchCoalescer expects 2"),
+ "unexpected error: {err}"
+ );
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_oob_before_mutation() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Utf8View, false),
+ ]));
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..16i32)),
+ Arc::new(StringViewArray::from(vec!["a"; 16])),
+ ],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1, 2, 3, 0, 1, 2, 99]);
Review Comment:
added in latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321707775
##########
arrow-select/src/coalesce/byte_view.rs:
##########
@@ -354,13 +362,157 @@ impl<B: ByteViewType> InProgressByteViewArray<B> {
self.views.extend(new_views);
self.current = Some(dst_buffer);
}
+
+ #[inline(never)]
+ fn append_indexed_views_and_copy_strings(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ ) {
+ self.append_indexed_views_and_copy_strings_with_max_buffer_size(
+ src_views,
+ indices,
+ src_buffers,
+ selected_bytes,
+ i32::MAX as usize,
+ );
+ }
+
+ fn append_indexed_views_and_copy_strings_with_max_buffer_size(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ max_buffer_size: usize,
+ ) {
+ if selected_bytes == 0 {
+ self.views.reserve(indices.len());
+ for &idx in indices {
+ // SAFETY: caller validated idx < src_views.len().
+ let raw = unsafe { *src_views.get_unchecked(idx as usize) };
+ self.views.push(raw);
+ }
+ return;
+ }
+
+ let mut dst_buffer = match self.current.take() {
+ Some(current)
+ if current.capacity() - current.len() >= selected_bytes
+ && current
+ .len()
+ .checked_add(selected_bytes)
+ .is_some_and(|len| len <= max_buffer_size) =>
+ {
+ current
+ }
+ Some(current) => {
+ let completed_buffer: Buffer = current.into();
+ self.completed_buffers_size += completed_buffer.capacity();
+ self.completed.push(completed_buffer);
+ self.buffer_source
+ .next_buffer(selected_bytes.min(max_buffer_size))
+ }
+ None => self
+ .buffer_source
+ .next_buffer(selected_bytes.min(max_buffer_size)),
+ };
+
+ let current_len = self.views.len();
+ self.views.reserve(indices.len());
+ let output = self.views.spare_capacity_mut().as_mut_ptr().cast::<u128>();
+ let mut copied_bytes = 0usize;
+ for (i, &idx) in indices.iter().enumerate() {
+ // SAFETY: caller validated idx < src_views.len().
+ let raw = unsafe { *src_views.get_unchecked(idx as usize) };
+ let mut view = ByteView::from(raw);
+ if view.length > MAX_INLINE_VIEW_LEN {
+ let buffer_index = view.buffer_index as usize;
+ let buffer_offset = view.offset as usize;
+ let str_len = view.length as usize;
+ assert!(str_len <= max_buffer_size);
+
+ let dst_end = dst_buffer
+ .len()
+ .checked_add(str_len)
+ .expect("byte view buffer size overflow");
+ if dst_end > max_buffer_size {
+ let completed_buffer: Buffer = dst_buffer.into();
+ self.completed_buffers_size += completed_buffer.capacity();
+ self.completed.push(completed_buffer);
+
+ let remaining = selected_bytes
+ .checked_sub(copied_bytes)
+ .expect("selected byte count mismatch");
+ dst_buffer = self
+ .buffer_source
+ .next_buffer(remaining.min(max_buffer_size));
+ }
+
+ // SAFETY: source views carry valid buffer_index/offset/length.
+ let src = unsafe {
+ src_buffers
+ .get_unchecked(buffer_index)
+ .get_unchecked(buffer_offset..buffer_offset + str_len)
+ };
+ view.offset = dst_buffer
+ .len()
+ .try_into()
+ .expect("byte view offset exceeds u32::MAX");
+ view.buffer_index = self.completed.len().try_into().expect("too many buffers");
+ let dst_len = dst_buffer.len();
+ // SAFETY: the destination was sized for the remaining selected bytes,
+ // and `src` belongs to the input array and cannot overlap it.
+ unsafe {
+ std::ptr::copy_nonoverlapping(
+ src.as_ptr(),
+ dst_buffer.as_mut_ptr().add(dst_len),
+ str_len,
+ );
+ dst_buffer.set_len(dst_len + str_len);
+ }
Review Comment:
will swap to use `extend_from_slice` perf should be pretty much the same and we avoid an unsafe block 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5886436857
🤔 on second thought. the new methods are useful for the current implementation but maybe introducing some buffer could be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5787108724
at the risk of scope creep I think its worth introducing a couple things in this PR
1. two new methods on the `inprogressArrays` trait that will make it easier to avoid copies with `push_interleave()`/`push_take()`
2. 1 new method `push_interleave()` in batchCoalescer
3. new benchmarks for this new method on batchCoalescer
follow up PR will implement
1. optimized `push_take` for primitives
2. optimized `push_interleave` for primitves
3. optimized `push_take` for utf8View
4. optimized `push_interleave` for utf8View
... fsb same idea
we will keep the default path for non-specialized arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179339608
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179337553)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (4850db63ea2da3304d698c9a905c47fcb9c368a0) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4850db63ea2da3304d698c9a905c47fcb9c368a0)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6179337553:start:8a54147f211b3527218ee8a5d52f28ae -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6180161846
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179337553)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (4850db63ea2da3304d698c9a905c47fcb9c368a0) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4850db63ea2da3304d698c9a905c47fcb9c368a0)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.79 13.7±0.13ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.53 3.4±0.06ms ? ?/sec 1.00 973.9±8.06µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 5.39 4.5±0.32ms ? ?/sec 1.00 826.2±3.92µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 6.92 3.3±0.34ms ? ?/sec 1.00 482.4±4.16µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.8±0.09ms ? ?/sec 1.00 15.4±0.08ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.80 4.9±0.11ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 3.85 5.3±0.38ms ? ?/sec 1.00 1385.5±5.92µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.04 885.9±9.67µs ? ?/sec 1.00 855.9±6.89µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.34 12.8±0.12ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.52 2.3±0.01ms ? ?/sec 1.00 923.4±5.27µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.00 1645.9±10.80µs ? ?/sec 1.00 822.9±2.65µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.44 1149.7±11.97µs ? ?/sec 1.00 471.2±3.27µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.8±0.10ms ? ?/sec 1.00 14.7±0.07ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.38 3.6±0.04ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.45 2.0±0.04ms ? ?/sec 1.00 1381.7±9.99µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.03 876.9±9.09µs ? ?/sec 1.00 847.8±8.64µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 123.3±4.53ms ? ?/sec 1.12 138.6±2.92ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.4±0.06ms ? ?/sec 1.16 6.3±0.25ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.9±0.06ms ? ?/sec 1.11 4.3±0.09ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.2±0.04ms ? ?/sec 1.05 2.3±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 127.7±1.79ms ? ?/sec 1.02 130.8±2.58ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.03 7.2±0.32ms ? ?/sec 1.00 7.0±0.24ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.9±0.06ms ? ?/sec 1.04 4.1±0.06ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.3±0.04ms ? ?/sec 1.02 2.4±0.02ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 205.4±4.48ms ? ?/sec 1.01 207.8±3.35ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.7±0.16ms ? ?/sec 1.15 26.0±0.77ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 6.6±0.09ms ? ?/sec 1.16 7.7±0.21ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 8.7±0.32ms ? ?/sec 1.07 9.3±0.57ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 7.8±0.18ms ? ?/sec 1.10 8.5±0.29ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 36.2±0.54ms ? ?/sec 1.01 36.6±0.71ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 9.3±0.31ms ? ?/sec 1.06 9.9±0.21ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.01 8.9±0.30ms ? ?/sec 1.00 8.9±0.44ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 7.0±0.32ms ? ?/sec 1.10 7.7±0.16ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.79 13.5±0.11ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.44 3.3±0.02ms ? ?/sec 1.00 968.3±7.29µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 5.25 4.4±0.29ms ? ?/sec 1.00 831.3±4.00µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 7.19 3.4±0.28ms ? ?/sec 1.00 472.2±2.42µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.8±0.11ms ? ?/sec 1.00 15.3±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.73 4.7±0.05ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 3.83 5.3±0.33ms ? ?/sec 1.00 1370.9±10.53µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.03 872.0±8.59µs ? ?/sec 1.00 849.0±6.43µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.75 12.8±0.12ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.59 2.2±0.01ms ? ?/sec 1.00 865.1±10.53µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.62 1314.8±13.95µs ? ?/sec 1.00 810.9±3.23µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.22 569.9±2.61µs ? ?/sec 1.00 468.5±3.02µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.36 19.9±0.10ms ? ?/sec 1.00 14.6±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.35 3.6±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.29 1793.0±21.42µs ? ?/sec 1.00 1391.2±6.29µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.02 864.3±7.90µs ? ?/sec 1.00 849.8±7.25µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 19.1±0.50ms ? ?/sec 1.04 19.9±0.49ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.21 30.4±0.38ms ? ?/sec 1.00 4.9±0.03ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.46 4.7±0.05ms ? ?/sec 1.00 1898.8±77.16µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.17 1659.0±5.19µs ? ?/sec 1.00 1412.6±4.50µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 829.0±1.39µs ? ?/sec 1.03 850.9±2.25µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.43 48.3±0.31ms ? ?/sec 1.00 33.9±0.21ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.16 7.5±0.04ms ? ?/sec 1.00 6.4±0.07ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.6±0.01ms ? ?/sec 1.02 2.7±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1541.7±7.56µs ? ?/sec 1.13 1734.8±10.21µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.7±0.25ms ? ?/sec 1.03 24.4±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.03ms ? ?/sec 1.05 5.4±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.05 3.0±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.03 2.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.3±0.24ms ? ?/sec 1.00 33.5±0.14ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.8±0.04ms ? ?/sec 1.04 7.1±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.07 3.2±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.02 2.3±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.68 13.6±0.25ms ? ?/sec 1.00 5.1±0.06ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.83 3.0±0.02ms ? ?/sec 1.00 1062.5±22.07µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.30 3.1±0.02ms ? ?/sec 1.00 950.8±4.23µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.27 807.5±1.50µs ? ?/sec 1.00 636.6±2.12µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.30 20.2±0.16ms ? ?/sec 1.00 15.5±0.10ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.73 4.2±0.02ms ? ?/sec 1.00 2.4±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.41 3.4±0.02ms ? ?/sec 1.00 1426.2±5.70µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.11 1055.9±2.50µs ? ?/sec 1.00 953.5±4.77µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.12ms ? ?/sec 1.02 13.9±0.09ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1551.4±8.62µs ? ?/sec 1.02 1584.0±6.78µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 359.9±0.99µs ? ?/sec 1.01 362.0±1.36µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 241.3±0.42µs ? ?/sec 1.00 241.8±0.62µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.3±0.13ms ? ?/sec 1.01 20.5±0.09ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.01 2.6±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 616.6±1.43µs ? ?/sec 1.02 628.3±1.54µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 435.4±0.79µs ? ?/sec 1.00 436.2±0.93µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.14ms ? ?/sec 1.01 12.7±0.05ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1712.3±13.00µs ? ?/sec 1.04 1782.8±14.26µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 714.5±1.77µs ? ?/sec 1.05 748.5±2.41µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 459.5±1.10µs ? ?/sec 1.03 471.9±2.50µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.5±0.09ms ? ?/sec 1.01 19.6±0.10ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1058.0±2.92µs ? ?/sec 1.04 1097.9±4.58µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 700.8±2.22µs ? ?/sec 1.06 740.4±2.57µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.1±0.12ms ? ?/sec 1.01 13.3±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.10 2.6±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 1952.5±10.14µs ? ?/sec 1.13 2.2±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1492.1±13.01µs ? ?/sec 1.11 1651.2±28.61µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.9±0.13ms ? ?/sec 1.00 20.9±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.03ms ? ?/sec 1.06 4.0±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.03ms ? ?/sec 1.13 2.7±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1802.2±25.83µs ? ?/sec 1.08 1952.6±19.93µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.9±0.21ms ? ?/sec 1.02 35.5±0.28ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.04 5.2±0.04ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1948.8±5.27µs ? ?/sec 1.06 2.1±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1201.6±3.79µs ? ?/sec 1.03 1240.6±4.92µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.9±0.17ms ? ?/sec 1.02 60.3±0.30ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.0±0.05ms ? ?/sec 1.01 11.1±0.05ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.4±0.02ms ? ?/sec 1.03 5.5±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.8±0.05ms ? ?/sec 1.01 4.8±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.2±0.34ms ? ?/sec 1.04 45.9±0.25ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.4±0.19ms ? ?/sec 1.09 10.2±0.06ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.06ms ? ?/sec 1.13 5.8±0.09ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.8±0.03ms ? ?/sec 1.05 4.0±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.8±0.32ms ? ?/sec 1.04 53.8±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.22ms ? ?/sec 1.06 11.6±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.1±0.05ms ? ?/sec 1.12 6.8±0.07ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.7±0.16ms ? ?/sec 1.05 4.9±0.13ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 35.6±0.19ms ? ?/sec 1.11 39.7±0.50ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.7±0.17ms ? ?/sec 1.15 14.6±0.30ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 17.7±1.07ms ? ?/sec 1.19 21.1±0.88ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 17.8±0.64ms ? ?/sec 1.13 20.1±1.00ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 48.5±0.61ms ? ?/sec 1.05 51.0±1.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 15.2±0.35ms ? ?/sec 1.05 16.0±0.24ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 17.8±0.67ms ? ?/sec 1.01 18.0±0.96ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 16.2±0.57ms ? ?/sec 1.12 18.1±0.45ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.69 13.3±0.18ms ? ?/sec 1.00 4.9±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.82 3.0±0.02ms ? ?/sec 1.00 1046.3±9.65µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 2.87 2.6±0.01ms ? ?/sec 1.00 905.7±2.09µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.26 805.8±1.45µs ? ?/sec 1.00 640.3±2.10µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.33 20.4±0.10ms ? ?/sec 1.00 15.3±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.75 4.2±0.03ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.99 2.8±0.01ms ? ?/sec 1.00 1396.5±4.75µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.12 1058.5±3.27µs ? ?/sec 1.00 943.5±3.35µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1smkhpuy-d95a44c1-bw5n
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6179337553-3416-8746b 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1400.3s |
| Peak memory | 60.3 MiB |
| Avg memory | 20.9 MiB |
| CPU user | 1372.7s |
| CPU sys | 25.7s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1460.3s |
| Peak memory | 60.0 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1427.1s |
| CPU sys | 27.8s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6179337553:finish:f0c887df4297ad5fdb3b31fef7b2fdd6 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4254733111
##########
arrow-select/src/coalesce.rs:
##########
@@ -335,11 +336,65 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
Review Comment:
thinking of depricating this in favor of a more flexible api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6275014834
> > @alamb, @mbutrovich this is ready for review, I cant mark it as ready for review since I have 3 other PR's open 😅
>
> Which one should I review first? i'll try and get one in
@alamb I have 4 open PR's two are approved [ #11407, #11519] two un-approved [#11462 , #11464 ]. could we merge the two approved ones
Id prefer if you reviewed this one first since its the most important to get shipped with the next release 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225831258
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225096087)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.18 13.5±0.10ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.68 3.3±0.03ms ? ?/sec 1.00 1981.9±18.01µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.06 3.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1965.0±11.61µs ? ?/sec 1.22 2.4±0.85ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.33 20.4±0.04ms ? ?/sec 1.00 15.3±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.19 4.5±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.01ms ? ?/sec 1.12 3.4±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.01 869.2±0.83µs ? ?/sec 1.00 860.9±1.17µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.86 12.7±0.12ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 1.97 2.3±0.02ms ? ?/sec 1.00 1191.2±28.90µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1673.5±2.80µs ? ?/sec 1.00 1666.4±10.76µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1103.2±5.06µs ? ?/sec 1.01 1117.9±4.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.6±0.09ms ? ?/sec 1.00 14.2±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.18 3.6±0.01ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.0±0.00ms ? ?/sec 1.05 2.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 858.3±1.16µs ? ?/sec 1.00 856.9±1.76µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 95.3±0.76ms ? ?/sec 1.00 95.1±0.65ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.00 5.5±0.01ms ? ?/sec 1.01 5.6±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.02 3.8±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 93.9±0.55ms ? ?/sec 1.00 94.3±0.76ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.8±0.01ms ? ?/sec 1.01 5.8±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.02 3.7±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 148.8±1.38ms ? ?/sec 1.00 148.7±1.18ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.02 23.0±0.10ms ? ?/sec 1.00 22.7±0.14ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 6.6±0.01ms ? ?/sec 1.01 6.7±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.03ms ? ?/sec 1.01 5.2±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.3±0.04ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.7±0.12ms ? ?/sec 1.00 32.9±0.12ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.1±0.02ms ? ?/sec 1.01 8.2±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.1±0.02ms ? ?/sec 1.01 5.2±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.0±0.03ms ? ?/sec 1.01 4.0±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.25 13.3±0.08ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.73 3.3±0.03ms ? ?/sec 1.00 1923.8±10.66µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1935.8±8.47µs ? ?/sec 1.00 1929.2±4.92µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.3±0.08ms ? ?/sec 1.00 15.2±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.19 4.4±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.01ms ? ?/sec 1.03 3.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 861.9±1.27µs ? ?/sec 1.00 861.5±1.21µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.95 12.6±0.06ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.12 2.3±0.02ms ? ?/sec 1.00 1102.5±9.66µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.04 1396.1±4.50µs ? ?/sec 1.00 1348.8±2.41µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.01 591.3±3.35µs ? ?/sec 1.00 586.7±1.09µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.5±0.07ms ? ?/sec 1.00 14.3±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.22 3.5±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1758.6±4.15µs ? ?/sec 1.05 1855.1±4.84µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.01 863.8±1.10µs ? ?/sec 1.00 855.5±0.92µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 12.3±0.09ms ? ?/sec 1.00 12.3±0.05ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 5.86 30.1±0.24ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.48 4.3±0.04ms ? ?/sec 1.00 1717.4±96.31µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.36 1759.1±2.21µs ? ?/sec 1.00 1293.6±1.95µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.01 842.9±3.10µs ? ?/sec 1.00 836.6±1.28µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.39 47.2±0.12ms ? ?/sec 1.00 34.0±0.13ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.11 7.0±0.02ms ? ?/sec 1.00 6.3±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.08 2.7±0.00ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1535.8±2.33µs ? ?/sec 1.00 1530.9±1.81µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.4±0.08ms ? ?/sec 1.01 23.6±0.12ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.02ms ? ?/sec 1.02 5.3±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 3.0±0.00ms ? ?/sec 1.00 3.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.16ms ? ?/sec 1.01 33.2±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.3±0.04ms ? ?/sec 1.09 6.9±0.12ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.1±0.02ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.99 13.3±0.17ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.38 2.9±0.02ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.3±0.01ms ? ?/sec 1.16 3.8±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 836.1±4.01µs ? ?/sec 1.00 839.6±2.04µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.31 20.2±0.10ms ? ?/sec 1.00 15.4±0.06ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.14 4.2±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.6±0.01ms ? ?/sec 1.16 4.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.02 1078.5±1.60µs ? ?/sec 1.00 1057.4±1.25µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.5±0.08ms ? ?/sec 1.03 13.9±0.18ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1557.6±4.36µs ? ?/sec 1.02 1592.9±16.07µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 363.2±0.56µs ? ?/sec 1.00 362.7±1.47µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.7±0.72µs ? ?/sec 1.00 239.8±0.38µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.4±0.10ms ? ?/sec 1.01 20.6±0.14ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.02 2.6±0.02ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 617.9±1.03µs ? ?/sec 1.02 629.6±1.67µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 430.0±0.98µs ? ?/sec 1.01 436.1±0.56µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.7±0.03ms ? ?/sec 1.00 12.7±0.20ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1748.7±3.80µs ? ?/sec 1.02 1776.3±31.46µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.03 760.5±1.78µs ? ?/sec 1.00 737.7±1.74µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 470.5±2.31µs ? ?/sec 1.00 472.2±0.46µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.3±0.12ms ? ?/sec 1.01 19.4±0.05ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.04 1090.6±2.22µs ? ?/sec 1.00 1043.7±1.52µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.02 711.2±2.03µs ? ?/sec 1.00 700.0±0.95µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.06ms ? ?/sec 1.02 13.3±0.14ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.01 2.5±0.01ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 2.1±0.01ms ? ?/sec 1.01 2.1±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1515.3±5.75µs ? ?/sec 1.02 1547.8±2.66µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.7±0.04ms ? ?/sec 1.00 20.7±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.01ms ? ?/sec 1.01 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.00ms ? ?/sec 1.01 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1771.9±4.16µs ? ?/sec 1.01 1783.4±3.48µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.8±0.18ms ? ?/sec 1.02 35.4±0.42ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.02 5.1±0.04ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1961.8±9.60µs ? ?/sec 1.03 2.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1242.0±5.22µs ? ?/sec 1.00 1239.8±2.16µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.7±0.14ms ? ?/sec 1.03 60.2±0.43ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.01ms ? ?/sec 1.01 11.0±0.05ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.03 5.4±0.00ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.6±0.23ms ? ?/sec 1.01 45.1±0.09ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.2±0.12ms ? ?/sec 1.01 9.3±0.20ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.4±0.02ms ? ?/sec 1.00 5.3±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.7±0.10ms ? ?/sec 1.01 52.5±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.05ms ? ?/sec 1.01 10.9±0.06ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.2±0.02ms ? ?/sec 1.00 6.2±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.02ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.2±0.17ms ? ?/sec 1.01 36.6±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.7±0.10ms ? ?/sec 1.00 12.6±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.3±0.05ms ? ?/sec 1.01 11.4±0.06ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.4±0.10ms ? ?/sec 1.01 10.4±0.12ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 47.0±0.26ms ? ?/sec 1.01 47.7±0.17ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.1±0.03ms ? ?/sec 1.01 14.2±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.6±0.06ms ? ?/sec 1.00 10.6±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 8.8±0.09ms ? ?/sec 1.00 8.9±0.07ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.02 13.2±0.12ms ? ?/sec 1.00 4.4±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.58 3.0±0.02ms ? ?/sec 1.00 1871.7±5.69µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.15 3.1±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.01 834.3±3.44µs ? ?/sec 1.00 824.0±1.05µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.35 19.9±0.14ms ? ?/sec 1.00 14.8±0.07ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.26 4.1±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.01ms ? ?/sec 1.16 3.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.02 1082.6±2.36µs ? ?/sec 1.00 1062.6±1.41µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-nwpa3evy-99aa7c2b-2zd5
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6225096087-3525-bvj4j 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1410.3s |
| Peak memory | 60.7 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 1380.2s |
| CPU sys | 23.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1400.3s |
| Peak memory | 61.5 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1370.4s |
| CPU sys | 23.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6225096087:finish:cd75371d1a393190a019b3a1d9948455 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6225096087
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4325727027
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
Review Comment:
> these traits and methods are private so users cant call this code arbitrarily. their respective conditions are outlined as safety comments for both `primitve` and `View` types. I think this method should remain safe
That's fair. `copy_rows` in this trait already works the same way: its impls call `get_unchecked` behind a `SAFETY` comment that relies on the caller ([primitive.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/primitive.rs#L167-L169), [byte_view.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/byte_view.rs#L411-L412)). I'm fine keeping `copy_rows_by_indices` safe.
The difference I see is where the guarantee comes from. `copy_rows` gets ranges that the coalescer computed itself. Here the indices are caller input, and the check that makes the reads safe lives in `push_batch_with_indices`, two calls up and in another file. That check also moved in the last commit, from `push_batch_with_index` up to its caller.
Could we add a `debug_assert!` next to each `SAFETY` comment that relies on caller validation? Those are in [`InProgressPrimitiveArray::copy_rows_by_indices`](https://github.com/apache/arrow-rs/blob/e31420bf3b6f3959e2a083bfdfa7dc68376580c3/arrow-select/src/coalesce/primitive.rs#L230-L243), [`InProgressByteViewArray::copy_rows_by_indices`](https://github.com/apache/arrow-rs/blob/e31420bf3b6f3959e2a083bfdfa7dc68376580c3/arrow-select/src/coalesce/byte_view.rs#L657-L665) (and [again here](https://github.com/apache/arrow-rs/blob/e31420bf3b6f3959e2a083bfdfa7dc68376580c3/arrow-select/src/coalesce/byte_view.rs#L714-L715)), [`selected_non_inline_bytes`](https://github.com/apache/arrow-rs/blob/e31420bf3b6f3959e2a083bfdfa7dc68376580c3/arrow-select/src/coalesce/byte_view.rs#L480-L488), and [`append_indexed_views_and_copy_strings_with_max_buffer_size`](https://github.com/apache/arrow-rs/blob/e31420bf3b6f3959e2a083bfdfa7dc68376580c3/arrow-select/src/coalesce/byte_view.rs#L383-L395). The unit te
sts in `primitive.rs` and `byte_view.rs` call `copy_rows_by_indices` directly, so they never pass through the assert in `push_batch_with_index`. Local asserts would also cover new callers, such as the `push_interleave` impls you mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6293289189
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292219011)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (e31420bf3b6f3959e2a083bfdfa7dc68376580c3) to 1357f25 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1357f25f26ca67634e992bdb35220b4ee59c780f..e31420bf3b6f3959e2a083bfdfa7dc68376580c3)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.13 13.5±0.12ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.47 3.3±0.01ms ? ?/sec 1.00 2.2±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.02ms ? ?/sec 1.29 3.5±0.16ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.4±0.08ms ? ?/sec 1.22 3.0±0.15ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 2.32 20.5±0.08ms ? ?/sec 1.00 8.9±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.43 4.6±0.04ms ? ?/sec 1.00 3.2±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 4.1±0.15ms ? ?/sec 1.22 5.0±0.21ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 866.5±4.68µs ? ?/sec 1.04 900.3±3.32µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.83 12.8±0.10ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 1.93 2.2±0.01ms ? ?/sec 1.00 1161.7±3.81µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1654.3±5.14µs ? ?/sec 1.00 1651.0±6.13µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1117.1±6.30µs ? ?/sec 1.02 1135.2±8.82µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 2.41 19.6±0.08ms ? ?/sec 1.00 8.2±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.71 3.5±0.02ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.0±0.01ms ? ?/sec 1.06 2.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 858.8±2.39µs ? ?/sec 1.05 900.5±3.25µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 97.8±1.53ms ? ?/sec 1.15 112.6±1.17ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.04 5.3±0.01ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.7±0.03ms ? ?/sec 1.04 3.9±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.09 2.2±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 117.5±1.45ms ? ?/sec 1.01 118.9±1.90ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.9±0.06ms ? ?/sec 1.05 6.1±0.10ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.03ms ? ?/sec 1.07 4.0±0.04ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.1±0.02ms ? ?/sec 1.05 2.2±0.03ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 184.3±2.72ms ? ?/sec 1.07 198.1±2.27ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.48 22.6±0.08ms ? ?/sec 1.00 15.3±0.08ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.12 6.6±0.06ms ? ?/sec 1.00 5.9±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.4±0.21ms ? ?/sec 1.28 6.9±0.43ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 5.9±0.34ms ? ?/sec 1.23 7.3±0.15ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.26 34.1±0.33ms ? ?/sec 1.00 27.1±0.45ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.02 8.3±0.08ms ? ?/sec 1.00 8.2±0.10ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 7.3±0.20ms ? ?/sec 1.15 8.3±0.19ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 6.2±0.15ms ? ?/sec 1.10 6.9±0.14ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.24 13.4±0.10ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.56 3.2±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.18ms ? ?/sec 1.23 3.4±0.14ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.3±0.21ms ? ?/sec 1.31 3.0±0.17ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 2.31 20.5±0.07ms ? ?/sec 1.00 8.9±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.47 4.5±0.02ms ? ?/sec 1.00 3.1±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 4.0±0.12ms ? ?/sec 1.21 4.8±0.19ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 860.6±3.40µs ? ?/sec 1.05 900.7±3.03µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.96 12.6±0.10ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.01 2.2±0.01ms ? ?/sec 1.00 1090.9±5.15µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.02 1361.1±1.89µs ? ?/sec 1.00 1339.3±4.48µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 577.6±0.96µs ? ?/sec 1.02 589.5±2.39µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 2.43 19.7±0.07ms ? ?/sec 1.00 8.1±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.73 3.5±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1717.3±5.66µs ? ?/sec 1.07 1836.5±8.80µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 859.8±1.90µs ? ?/sec 1.05 907.1±6.89µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 14.7±0.66ms ? ?/sec 1.03 15.2±0.30ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.17 29.9±0.25ms ? ?/sec 1.00 4.9±0.06ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.81 4.2±0.03ms ? ?/sec 1.00 1494.7±91.33µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.39 1671.2±4.75µs ? ?/sec 1.00 1198.7±2.20µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 838.3±1.69µs ? ?/sec 1.03 866.8±2.04µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 2.58 47.5±0.17ms ? ?/sec 1.00 18.4±0.05ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.77 6.8±0.02ms ? ?/sec 1.00 3.9±0.07ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.22 2.6±0.01ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1529.6±2.57µs ? ?/sec 1.04 1590.5±3.40µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.06 23.5±0.16ms ? ?/sec 1.00 22.1±0.13ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.01 5.1±0.03ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.03 2.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.02 32.9±0.11ms ? ?/sec 1.00 32.2±0.11ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.3±0.07ms ? ?/sec 1.09 6.8±0.11ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.03ms ? ?/sec 1.07 3.1±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.03ms ? ?/sec 1.05 2.3±0.10ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.16 13.7±0.21ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.42 2.9±0.04ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 808.5±0.88µs ? ?/sec 1.08 869.6±1.90µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 2.27 20.2±0.20ms ? ?/sec 1.00 8.9±0.05ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.52 4.1±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.08 3.7±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1049.0±1.91µs ? ?/sec 1.08 1129.8±2.77µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.09 14.0±0.36ms ? ?/sec 1.00 12.8±0.18ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.07 1603.2±47.50µs ? ?/sec 1.00 1495.4±19.18µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 362.9±4.49µs ? ?/sec 1.11 401.4±2.04µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.0±0.60µs ? ?/sec 1.16 278.3±0.64µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.07 20.7±0.38ms ? ?/sec 1.00 19.4±0.05ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.04 2.6±0.04ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 621.1±4.23µs ? ?/sec 1.07 661.6±0.64µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 433.7±0.75µs ? ?/sec 1.08 469.9±0.82µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.14 12.3±0.17ms ? ?/sec 1.00 10.8±0.10ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.01 1659.1±10.64µs ? ?/sec 1.00 1639.3±17.68µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 692.9±1.60µs ? ?/sec 1.07 740.3±2.68µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 448.5±0.61µs ? ?/sec 1.12 504.4±0.98µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.08 19.1±0.13ms ? ?/sec 1.00 17.8±0.19ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1013.2±1.56µs ? ?/sec 1.10 1112.9±4.19µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 682.8±1.57µs ? ?/sec 1.12 763.4±1.69µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.18 13.2±0.28ms ? ?/sec 1.00 11.2±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.03 2.4±0.01ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.01 2.0±0.00ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1468.2±3.05µs ? ?/sec 1.07 1573.9±8.09µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.07 20.4±0.09ms ? ?/sec 1.00 19.0±0.19ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.01 3.7±0.02ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.01ms ? ?/sec 1.12 2.6±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1710.9±5.46µs ? ?/sec 1.13 1927.7±20.54µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.03 34.6±0.31ms ? ?/sec 1.00 33.7±0.15ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.03ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1915.9±4.65µs ? ?/sec 1.02 1956.8±7.57µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1180.4±2.47µs ? ?/sec 1.13 1328.4±6.48µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.5±0.21ms ? ?/sec 1.00 58.4±0.22ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.5±0.03ms ? ?/sec 1.01 10.6±0.04ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 4.8±0.01ms ? ?/sec 1.06 5.1±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.01ms ? ?/sec 1.06 4.5±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.01 44.0±0.19ms ? ?/sec 1.00 43.5±0.21ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.01 9.4±0.19ms ? ?/sec 1.00 9.3±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.01ms ? ?/sec 1.02 5.4±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.06 3.9±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.8±0.28ms ? ?/sec 1.00 51.5±0.25ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.02 10.7±0.35ms ? ?/sec 1.00 10.5±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.1±0.36ms ? ?/sec 1.05 6.4±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.34ms ? ?/sec 1.04 4.8±0.06ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.04 36.2±0.17ms ? ?/sec 1.00 34.8±0.21ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.3±0.04ms ? ?/sec 1.04 12.7±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.6±0.22ms ? ?/sec 1.43 16.6±0.67ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 14.8±0.46ms ? ?/sec 1.19 17.6±0.46ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 47.3±0.19ms ? ?/sec 1.06 49.9±0.41ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.5±0.11ms ? ?/sec 1.08 15.6±0.15ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 14.6±0.75ms ? ?/sec 1.22 17.8±0.72ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 14.3±0.67ms ? ?/sec 1.15 16.3±0.64ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.05 13.3±0.14ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.44 2.9±0.01ms ? ?/sec 1.00 1979.9±17.47µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.6±0.00ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 820.2±1.05µs ? ?/sec 1.07 874.3±2.07µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 2.28 20.0±0.08ms ? ?/sec 1.00 8.8±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.53 4.1±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.10 3.0±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1048.1±1.60µs ? ?/sec 1.08 1127.5±1.95µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-15ehpxtl-b2c6dea0-7qkl
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6292219011-3639-wnmvj 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 61.0 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1377.9s |
| CPU sys | 21.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 59.8 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1368.6s |
| CPU sys | 24.1s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6292219011:finish:5a45efb05801906fe330ad3ff322a544 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5819039279
PR description and commits are cleaned up. ready for review 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4077983805
##########
arrow/benches/coalesce_kernels.rs:
##########
@@ -611,9 +611,99 @@ fn add_all_take_benchmarks(c: &mut Criterion) {
}
}
-criterion_group!(benches, add_all_filter_benchmarks, add_all_take_benchmarks);
+criterion_group!(
+ benches,
+ add_all_filter_benchmarks,
+ add_all_take_benchmarks,
+ add_all_interleave_benchmarks
+);
criterion_main!(benches);
+fn add_all_interleave_benchmarks(c: &mut Criterion) {
+ let batch_size = 8192;
+ let num_source_batches = 4;
+
+ let primitive_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("int32_val", DataType::Int32, true),
+ Field::new("float_val", DataType::Float64, true),
+ Field::new(
+ "timestamp_val",
+ DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into())),
+ true,
+ ),
+ ]));
+
+ let single_utf8view_schema = SchemaRef::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Utf8View,
+ true,
+ )]));
+
+ let mixed_fsb_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("fsb16_val", DataType::FixedSizeBinary(16), true),
+ Field::new("fsb32_val", DataType::FixedSizeBinary(32), true),
+ ]));
+
+ // Binary, boolean, and dictionary all go through the generic path; group
+ // them in one schema so a single benchmark covers the whole family.
+ let mixed_generic_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("binary_val", DataType::Binary, true),
+ Field::new("bool_val", DataType::Boolean, true),
+ Field::new(
+ "dict_val",
+ DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8)),
+ true,
+ ),
+ ]));
+
+ for null_density in [0.0, 0.1] {
+ for selectivity in [0.001, 0.01, 0.1, 0.8] {
+ for scenario in [
Review Comment:
yea seems `push_batch_with_indices` already does this `take_streams`. will refactor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176520201
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226707444
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226706836)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6226706836:start:1bdb1480dd08e22cebc4b324a7e8e527 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175114491
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175113536)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a766c827931e2b8132df458b3f4ed7aafe918303) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a766c827931e2b8132df458b3f4ed7aafe918303)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175113536:start:753b40a79e070f5c651a70fd52967fd3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6180183900
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6179339186)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (4850db63ea2da3304d698c9a905c47fcb9c368a0) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4850db63ea2da3304d698c9a905c47fcb9c368a0)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.81 13.6±0.09ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.61 3.3±0.01ms ? ?/sec 1.00 923.7±6.95µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.44 2.8±0.03ms ? ?/sec 1.00 801.6±0.98µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.26 1958.3±7.76µs ? ?/sec 1.00 459.9±0.59µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.4±0.04ms ? ?/sec 1.00 15.2±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.75 4.5±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.40 3.0±0.03ms ? ?/sec 1.00 1266.3±2.48µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.08 863.3±0.93µs ? ?/sec 1.00 799.1±1.49µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.36 12.6±0.08ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.72 2.3±0.01ms ? ?/sec 1.00 857.3±7.92µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.08 1636.9±3.37µs ? ?/sec 1.00 785.5±0.67µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.43 1104.0±4.34µs ? ?/sec 1.00 453.9±0.45µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.6±0.03ms ? ?/sec 1.00 14.5±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.45 3.6±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.55 1986.4±2.05µs ? ?/sec 1.00 1283.6±2.86µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 857.0±1.24µs ? ?/sec 1.00 794.4±2.30µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 105.0±1.30ms ? ?/sec 1.01 106.3±1.53ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.01 5.5±0.02ms ? ?/sec 1.00 5.4±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.8±0.02ms ? ?/sec 1.02 3.8±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.03 106.1±1.52ms ? ?/sec 1.00 103.0±1.50ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.02 5.8±0.03ms ? ?/sec 1.00 5.7±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.03ms ? ?/sec 1.07 3.9±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.1±0.02ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 152.7±2.81ms ? ?/sec 1.01 154.5±2.72ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.03 23.2±0.06ms ? ?/sec 1.00 22.6±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.01 6.6±0.02ms ? ?/sec 1.00 6.6±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.4±0.07ms ? ?/sec 1.01 5.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.4±0.07ms ? ?/sec 1.03 4.5±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.02 32.9±0.13ms ? ?/sec 1.00 32.1±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.01 8.2±0.02ms ? ?/sec 1.00 8.1±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.3±0.10ms ? ?/sec 1.00 5.2±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.1±0.10ms ? ?/sec 1.01 4.1±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.85 13.5±0.13ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.52 3.3±0.01ms ? ?/sec 1.00 933.2±3.18µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.31 2.6±0.02ms ? ?/sec 1.00 794.7±1.20µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.24 1927.0±5.04µs ? ?/sec 1.00 454.9±0.48µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.5±0.11ms ? ?/sec 1.00 15.1±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.74 4.4±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.26 2.9±0.01ms ? ?/sec 1.00 1274.2±2.34µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.06 855.3±0.60µs ? ?/sec 1.00 807.7±3.21µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.76 12.7±0.08ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.88 2.3±0.02ms ? ?/sec 1.00 800.9±4.04µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.75 1352.4±4.34µs ? ?/sec 1.00 774.1±0.82µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.23 574.0±0.74µs ? ?/sec 1.00 465.6±1.80µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.5±0.02ms ? ?/sec 1.00 14.5±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.43 3.5±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.34 1725.3±3.06µs ? ?/sec 1.00 1288.6±1.76µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 861.7±0.78µs ? ?/sec 1.00 800.2±2.54µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.01 12.0±0.11ms ? ?/sec 1.00 11.8±0.21ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.18 30.2±0.21ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.94 4.7±0.08ms ? ?/sec 1.00 1586.1±6.05µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.29 1713.4±2.80µs ? ?/sec 1.00 1332.3±1.68µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.01 829.0±2.82µs ? ?/sec 1.00 823.6±1.57µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.4±0.07ms ? ?/sec 1.00 33.7±0.13ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.16 7.1±0.19ms ? ?/sec 1.00 6.1±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.07 2.7±0.00ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1528.2±2.30µs ? ?/sec 1.09 1671.0±2.28µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.5±0.08ms ? ?/sec 1.01 23.8±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.01 5.2±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.01 2.9±0.00ms ? ?/sec 1.00 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.0±0.12ms ? ?/sec 1.00 33.0±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.8±0.04ms ? ?/sec 1.00 6.8±0.07ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.0±0.02ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.74 13.5±0.20ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.98 3.0±0.01ms ? ?/sec 1.00 1006.5±7.73µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.60 3.2±0.01ms ? ?/sec 1.00 889.2±1.92µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.34 811.6±0.63µs ? ?/sec 1.00 606.8±0.70µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.32 20.1±0.10ms ? ?/sec 1.00 15.2±0.05ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.73 4.1±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.69 3.5±0.00ms ? ?/sec 1.00 1298.8±3.79µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.17 1053.7±1.36µs ? ?/sec 1.00 903.7±0.77µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.09ms ? ?/sec 1.01 13.8±0.05ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1542.3±3.25µs ? ?/sec 1.02 1577.2±7.85µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 362.3±1.28µs ? ?/sec 1.00 362.4±1.05µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 241.0±0.24µs ? ?/sec 1.00 239.9±0.42µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.2±0.11ms ? ?/sec 1.01 20.4±0.06ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 614.1±1.15µs ? ?/sec 1.01 623.3±1.36µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.1±0.24µs ? ?/sec 1.01 435.0±0.35µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.7±0.22ms ? ?/sec 1.00 12.7±0.12ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1692.8±10.46µs ? ?/sec 1.02 1735.1±18.21µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 718.0±1.54µs ? ?/sec 1.04 744.4±2.61µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 460.1±0.65µs ? ?/sec 1.00 461.5±1.78µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.5±0.20ms ? ?/sec 1.00 19.5±0.16ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.01ms ? ?/sec 1.01 3.1±0.02ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.01 1050.7±1.25µs ? ?/sec 1.00 1043.6±2.92µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 693.0±1.11µs ? ?/sec 1.01 701.7±1.08µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 12.9±0.05ms ? ?/sec 1.04 13.4±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.02 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1509.9±4.39µs ? ?/sec 1.04 1563.1±7.10µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.5±0.04ms ? ?/sec 1.01 20.8±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.01ms ? ?/sec 1.02 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.00ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1744.2±3.62µs ? ?/sec 1.01 1767.1±6.80µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.5±0.13ms ? ?/sec 1.00 34.7±0.11ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1937.2±4.29µs ? ?/sec 1.01 1955.4±3.24µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1206.3±1.79µs ? ?/sec 1.00 1205.8±10.94µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.3±0.13ms ? ?/sec 1.00 58.2±0.14ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.8±0.01ms ? ?/sec 1.01 10.9±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.02 5.3±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.2±0.16ms ? ?/sec 1.00 44.3±0.11ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.7±0.09ms ? ?/sec 1.00 9.7±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.01 5.3±0.02ms ? ?/sec 1.00 5.2±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.6±0.16ms ? ?/sec 1.00 51.6±0.14ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.02 10.7±0.04ms ? ?/sec 1.00 10.5±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.02 6.1±0.02ms ? ?/sec 1.00 6.0±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.02 4.4±0.03ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.01 36.9±0.20ms ? ?/sec 1.00 36.4±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.01 12.6±0.03ms ? ?/sec 1.00 12.5±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.6±0.21ms ? ?/sec 1.00 11.7±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.7±0.21ms ? ?/sec 1.03 11.1±0.32ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.8±0.19ms ? ?/sec 1.01 47.2±0.16ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.9±0.04ms ? ?/sec 1.02 14.2±0.09ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.05 11.1±0.22ms ? ?/sec 1.00 10.6±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 8.9±0.15ms ? ?/sec 1.03 9.2±0.19ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.73 13.2±0.16ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.84 3.0±0.02ms ? ?/sec 1.00 1043.0±7.50µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 2.84 2.7±0.00ms ? ?/sec 1.00 948.6±1.29µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.30 812.7±0.73µs ? ?/sec 1.00 626.9±0.67µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.31 19.9±0.08ms ? ?/sec 1.00 15.3±0.10ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.68 4.0±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.23 2.8±0.01ms ? ?/sec 1.00 1278.4±1.91µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.17 1057.2±1.42µs ? ?/sec 1.00 905.7±1.51µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1smkhpuy-5f13b132-m8rm
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6179339186-3417-9gngb 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 60.3 MiB |
| Avg memory | 20.8 MiB |
| CPU user | 1382.3s |
| CPU sys | 21.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1435.3s |
| Peak memory | 60.9 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1407.6s |
| CPU sys | 21.1s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6179339186:finish:b3b30c0ddc68eba192ef77631a64aae2 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241925367
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216105)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (40874968ff2a73c735b384cd46f3bb24f191e46c) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..40874968ff2a73c735b384cd46f3bb24f191e46c)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.61 13.7±0.08ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.70 3.3±0.02ms ? ?/sec 1.00 1953.0±9.89µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.03 2.7±0.07ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.01 1949.6±15.02µs ? ?/sec 1.00 1923.6±21.15µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.4±0.06ms ? ?/sec 1.00 15.1±0.16ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.15 4.4±0.01ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.01ms ? ?/sec 1.07 3.2±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 852.9±1.18µs ? ?/sec 1.00 849.5±1.04µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.32 12.8±0.07ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.12 2.3±0.01ms ? ?/sec 1.00 1086.8±4.97µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.09 1623.6±3.74µs ? ?/sec 1.00 1486.9±8.18µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1104.0±4.12µs ? ?/sec 1.00 1104.9±7.10µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.6±0.04ms ? ?/sec 1.00 14.2±0.17ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.17 3.4±0.01ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1960.0±7.14µs ? ?/sec 1.02 1990.2±5.08µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.01 850.5±1.13µs ? ?/sec 1.00 846.1±4.23µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.00 98.1±1.75ms ? ?/sec 1.00 97.8±1.36ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.02 5.3±0.02ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.6±0.02ms ? ?/sec 1.03 3.8±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 95.7±1.55ms ? ?/sec 1.01 96.3±1.42ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.7±0.02ms ? ?/sec 1.03 5.9±0.06ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.03ms ? ?/sec 1.01 3.7±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.07 173.4±4.03ms ? ?/sec 1.00 161.9±5.54ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.5±0.10ms ? ?/sec 1.02 23.0±0.08ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.01 6.4±0.03ms ? ?/sec 1.00 6.3±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.08 5.7±0.38ms ? ?/sec 1.00 5.3±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.01 4.6±0.19ms ? ?/sec 1.00 4.6±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.3±0.09ms ? ?/sec 1.00 32.4±0.17ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 7.9±0.02ms ? ?/sec 1.04 8.2±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.2±0.06ms ? ?/sec 1.03 5.3±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.2±0.10ms ? ?/sec 1.00 4.2±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.73 13.7±0.10ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.74 3.3±0.03ms ? ?/sec 1.00 1873.0±8.97µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.14 2.6±0.11ms ? ?/sec 1.00 2.3±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1903.2±8.98µs ? ?/sec 1.00 1893.9±6.64µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.36 20.3±0.05ms ? ?/sec 1.00 15.0±0.17ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.18 4.4±0.01ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.02 2.9±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 848.8±1.28µs ? ?/sec 1.01 855.7±1.14µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.42 12.9±0.05ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.22 2.3±0.01ms ? ?/sec 1.00 1015.1±12.22µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.15 1340.1±4.28µs ? ?/sec 1.00 1167.9±2.34µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.01 574.9±0.74µs ? ?/sec 1.00 568.4±1.35µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.39 19.5±0.03ms ? ?/sec 1.00 14.0±0.18ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.15 3.4±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1702.0±4.37µs ? ?/sec 1.01 1720.1±3.02µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 845.4±1.07µs ? ?/sec 1.01 852.3±1.34µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.8±0.12ms ? ?/sec 1.05 12.4±0.49ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.24 30.2±0.21ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.62 4.4±0.14ms ? ?/sec 1.00 1673.0±65.08µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.37 1679.4±5.59µs ? ?/sec 1.00 1226.5±3.34µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.02 838.3±0.96µs ? ?/sec 1.00 821.4±1.56µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.2±0.11ms ? ?/sec 1.00 33.6±0.72ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.13 6.9±0.02ms ? ?/sec 1.00 6.1±0.10ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.06 2.6±0.00ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.01 1552.4±2.05µs ? ?/sec 1.00 1534.6±1.30µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.4±0.09ms ? ?/sec 1.03 24.2±0.44ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.02 2.9±0.00ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.8±0.08ms ? ?/sec 1.03 33.9±0.45ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.04 6.7±0.05ms ? ?/sec 1.00 6.4±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.03ms ? ?/sec 1.01 3.0±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.03ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.33 13.4±0.21ms ? ?/sec 1.00 4.0±0.03ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.61 2.9±0.01ms ? ?/sec 1.00 1818.8±4.11µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.2±0.01ms ? ?/sec 1.05 3.3±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 837.3±0.96µs ? ?/sec 1.00 839.4±1.25µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.39 20.0±0.10ms ? ?/sec 1.00 14.4±0.16ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.23 4.1±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.10 3.8±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1080.6±1.54µs ? ?/sec 1.01 1090.3±1.76µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.7±0.09ms ? ?/sec 1.03 14.1±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1559.4±7.29µs ? ?/sec 1.04 1621.5±11.41µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 359.7±0.56µs ? ?/sec 1.02 366.5±1.20µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 239.2±0.30µs ? ?/sec 1.01 240.9±0.31µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.2±0.09ms ? ?/sec 1.02 20.7±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.02 2.6±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 615.2±0.72µs ? ?/sec 1.02 628.9±1.19µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 430.1±0.57µs ? ?/sec 1.02 438.1±0.49µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.13ms ? ?/sec 1.02 12.8±0.23ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1744.3±16.08µs ? ?/sec 1.01 1762.2±19.80µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.04 711.1±1.70µs ? ?/sec 1.00 686.5±1.64µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.03 457.2±0.94µs ? ?/sec 1.00 443.2±0.57µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.6±0.11ms ? ?/sec 1.00 19.6±0.13ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.03 3.1±0.02ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1007.9±1.96µs ? ?/sec 1.01 1020.7±2.47µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.01 697.0±1.59µs ? ?/sec 1.00 688.6±0.94µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.02 13.6±0.27ms ? ?/sec 1.00 13.3±0.43ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.01 1991.4±2.13µs ? ?/sec 1.00 1970.0±4.44µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.01 1498.3±7.50µs ? ?/sec 1.00 1481.7±2.74µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.8±0.04ms ? ?/sec 1.01 21.1±0.14ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.8±0.01ms ? ?/sec 1.02 3.8±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.00ms ? ?/sec 1.04 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.01 1708.7±3.63µs ? ?/sec 1.00 1684.8±3.03µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.7±0.21ms ? ?/sec 1.00 34.7±0.13ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.04 1882.5±2.71µs ? ?/sec 1.00 1802.0±2.81µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.03 1214.1±1.94µs ? ?/sec 1.00 1174.4±1.46µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.2±0.08ms ? ?/sec 1.03 59.8±0.26ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.7±0.01ms ? ?/sec 1.00 10.7±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 4.9±0.01ms ? ?/sec 1.01 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.01ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 43.9±0.16ms ? ?/sec 1.02 44.7±0.31ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.06 9.6±0.06ms ? ?/sec 1.00 9.0±0.08ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.02ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.02 3.7±0.02ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.7±0.11ms ? ?/sec 1.02 52.7±0.35ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.5±0.03ms ? ?/sec 1.00 10.6±0.07ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.02ms ? ?/sec 1.03 6.1±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.03ms ? ?/sec 1.00 4.3±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 35.8±0.14ms ? ?/sec 1.02 36.6±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.04 12.6±0.10ms ? ?/sec 1.00 12.1±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.02 11.8±0.46ms ? ?/sec 1.00 11.5±0.15ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 11.1±0.26ms ? ?/sec 1.00 11.1±0.27ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.8±0.18ms ? ?/sec 1.00 47.0±0.22ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.8±0.04ms ? ?/sec 1.02 14.2±0.05ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.6±0.10ms ? ?/sec 1.03 10.9±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.01 9.5±0.22ms ? ?/sec 1.00 9.4±0.22ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.48 13.5±0.41ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.70 2.9±0.04ms ? ?/sec 1.00 1713.1±3.94µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.02 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 840.4±0.91µs ? ?/sec 1.00 837.8±1.07µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.44 20.4±0.33ms ? ?/sec 1.00 14.1±0.17ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.32 4.1±0.04ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.07 3.0±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1074.3±1.63µs ? ?/sec 1.02 1091.5±1.88µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1odthu7g-9d446299-bqvd
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6241216105-3553-94l95 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1410.3s |
| Peak memory | 60.4 MiB |
| Avg memory | 20.5 MiB |
| CPU user | 1383.2s |
| CPU sys | 21.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 59.6 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1378.2s |
| CPU sys | 21.5s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6241216105:finish:c3dc23b264bcb548af4b417f0975524e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6221334194
this is much faster for low selectivity but fails for high selectivity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292059412
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4325947029
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &UInt32Array,
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let taken = take(source.as_ref(), indices, None)?;
+ self.set_source(Some(taken));
+ let result = self.copy_rows(0, indices.len());
+ self.set_source(None);
+ result
Review Comment:
https://github.com/apache/arrow-rs/pull/11174/changes/aaa05b39e1150db732b37b0e4d5f8a6e860367d5 adds `test_push_batch_with_indices_mixed_schema_with_nulls_splits_matches_take` which covers non-specialized types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4326192187
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
Review Comment:
>Could we add a debug_assert! next to each SAFETY comment that relies on caller validation?
yes, adding this. also adding two test that error out on this case (`primitive`/`utf8View`). commit https://github.com/apache/arrow-rs/pull/11174/changes/44bbfcfdf9d918f4baca173a47ad4b7c7e383ff7 ;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6303116487
thank you for the review, https://github.com/apache/arrow-rs/pull/11174/changes/cec88d2c260ad9e78f0616b7f6356d95685100f6 should address your safety concerns @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175800199
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176090274
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175113536)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a766c827931e2b8132df458b3f4ed7aafe918303) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a766c827931e2b8132df458b3f4ed7aafe918303)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.87 13.5±0.08ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.73 3.4±0.03ms ? ?/sec 1.00 908.9±3.35µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 4.69 3.6±0.16ms ? ?/sec 1.00 758.0±1.94µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 5.68 2.4±0.13ms ? ?/sec 1.00 416.7±2.48µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.5±0.09ms ? ?/sec 1.00 15.1±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.82 4.7±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.87 3.6±0.13ms ? ?/sec 1.00 1263.9±2.55µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.13 877.6±4.89µs ? ?/sec 1.00 773.6±1.50µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.51 12.7±0.09ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.83 2.3±0.01ms ? ?/sec 1.00 818.3±3.68µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.30 1731.2±9.79µs ? ?/sec 1.00 752.0±1.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.67 1122.1±4.97µs ? ?/sec 1.00 420.4±1.61µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.6±0.07ms ? ?/sec 1.00 14.3±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.44 3.6±0.03ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.61 2.1±0.01ms ? ?/sec 1.00 1289.4±3.80µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.13 870.6±2.38µs ? ?/sec 1.00 770.6±1.41µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.01 116.6±3.15ms ? ?/sec 1.00 115.3±3.78ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.06 5.7±0.06ms ? ?/sec 1.00 5.3±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.05 3.9±0.04ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.03 116.6±3.10ms ? ?/sec 1.00 113.2±2.27ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.01 6.1±0.06ms ? ?/sec 1.00 6.1±0.10ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.03 3.9±0.03ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.1±0.02ms ? ?/sec 1.01 2.1±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 14.76 177.4±5.74ms ? ?/sec 1.00 12.0±0.19ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.16 23.0±0.11ms ? ?/sec 1.00 19.8±0.10ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.02 6.7±0.05ms ? ?/sec 1.00 6.5±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.03 6.6±0.34ms ? ?/sec 1.00 6.5±0.33ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.04 6.1±0.47ms ? ?/sec 1.00 5.9±0.56ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.3±0.24ms ? ?/sec 1.01 33.7±0.34ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.02 8.6±0.08ms ? ?/sec 1.00 8.5±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.4±0.28ms ? ?/sec 1.05 6.7±0.40ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 5.3±0.32ms ? ?/sec 1.08 5.7±0.31ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.86 13.3±0.06ms ? ?/sec 1.00 3.4±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.84 3.4±0.02ms ? ?/sec 1.00 879.0±4.03µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 4.25 3.2±0.20ms ? ?/sec 1.00 758.9±1.54µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 5.63 2.3±0.10ms ? ?/sec 1.00 412.7±1.03µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.36 20.5±0.07ms ? ?/sec 1.00 15.1±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.77 4.6±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.68 3.4±0.14ms ? ?/sec 1.00 1267.3±2.60µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.15 875.5±3.14µs ? ?/sec 1.00 762.9±1.75µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.90 12.6±0.05ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 3.00 2.2±0.01ms ? ?/sec 1.00 748.6±4.83µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.93 1437.4±6.10µs ? ?/sec 1.00 743.5±1.08µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.43 589.5±1.54µs ? ?/sec 1.00 411.9±1.74µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.7±0.09ms ? ?/sec 1.00 14.2±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.42 3.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.39 1791.0±7.77µs ? ?/sec 1.00 1284.6±2.40µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.14 874.1±4.79µs ? ?/sec 1.00 766.2±1.66µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 16.07 12.3±0.47ms ? ?/sec 1.00 763.1±10.25µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.43 30.1±0.19ms ? ?/sec 1.00 4.7±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.91 4.2±0.02ms ? ?/sec 1.00 1454.2±90.84µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.36 1743.4±3.27µs ? ?/sec 1.00 1279.1±1.42µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.18 846.9±2.13µs ? ?/sec 1.00 720.1±0.85µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.3±0.07ms ? ?/sec 1.00 33.5±0.12ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.10 6.9±0.02ms ? ?/sec 1.00 6.3±0.05ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.07 2.7±0.00ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1556.4±4.17µs ? ?/sec 1.01 1567.4±3.56µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.5±0.10ms ? ?/sec 1.23 28.8±0.13ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.02ms ? ?/sec 1.19 6.2±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 3.0±0.01ms ? ?/sec 1.04 3.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.00ms ? ?/sec 1.02 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.2±0.09ms ? ?/sec 1.17 38.9±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.6±0.16ms ? ?/sec 1.19 7.9±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.1±0.02ms ? ?/sec 1.08 3.3±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.06 2.4±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.77 13.4±0.21ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 3.37 3.0±0.01ms ? ?/sec 1.00 886.8±3.25µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 4.25 3.3±0.01ms ? ?/sec 1.00 779.6±2.46µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.76 831.0±1.80µs ? ?/sec 1.00 473.3±0.87µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.33 20.0±0.16ms ? ?/sec 1.00 15.0±0.05ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.85 4.2±0.02ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 3.00 3.6±0.01ms ? ?/sec 1.00 1201.2±2.80µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.31 1075.6±3.13µs ? ?/sec 1.00 818.4±1.92µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.7±0.07ms ? ?/sec 1.30 17.9±0.07ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1559.4±3.94µs ? ?/sec 1.36 2.1±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 361.0±0.75µs ? ?/sec 1.36 490.8±1.68µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.4±0.21µs ? ?/sec 1.22 292.9±0.37µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.4±0.11ms ? ?/sec 1.25 25.5±0.14ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.24 3.1±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 615.1±1.14µs ? ?/sec 1.26 777.8±1.01µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.3±0.38µs ? ?/sec 1.14 490.0±0.38µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.07ms ? ?/sec 1.33 16.7±0.13ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1731.4±6.52µs ? ?/sec 1.41 2.4±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 747.2±1.66µs ? ?/sec 1.17 877.2±1.32µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 478.4±0.83µs ? ?/sec 1.20 576.1±0.69µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.0±0.07ms ? ?/sec 1.24 23.6±0.10ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.02ms ? ?/sec 1.15 3.5±0.02ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1080.2±2.53µs ? ?/sec 1.14 1236.5±1.95µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 725.0±1.58µs ? ?/sec 1.14 828.2±1.48µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.11ms ? ?/sec 1.33 17.3±0.11ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.32 3.2±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 2.1±0.01ms ? ?/sec 1.05 2.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1577.5±7.81µs ? ?/sec 1.08 1699.1±8.99µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.6±0.08ms ? ?/sec 1.21 25.0±0.05ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.9±0.01ms ? ?/sec 1.12 4.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.5±0.01ms ? ?/sec 1.08 2.7±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1810.6±5.25µs ? ?/sec 1.11 2.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.8±0.14ms ? ?/sec 1.15 39.9±0.34ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.13 5.6±0.05ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1958.8±6.27µs ? ?/sec 1.11 2.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1259.8±5.32µs ? ?/sec 1.07 1349.1±3.01µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 59.4±0.17ms ? ?/sec 1.09 64.7±0.41ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.01ms ? ?/sec 1.06 11.5±0.05ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.4±0.01ms ? ?/sec 1.04 5.6±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.7±0.01ms ? ?/sec 1.04 4.8±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.8±0.15ms ? ?/sec 1.11 49.5±0.17ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.8±0.04ms ? ?/sec 1.03 10.1±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.4±0.02ms ? ?/sec 1.02 5.5±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.8±0.02ms ? ?/sec 1.03 3.9±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.5±0.18ms ? ?/sec 1.09 57.2±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.3±0.26ms ? ?/sec 1.04 11.7±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.3±0.03ms ? ?/sec 1.02 6.5±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.03ms ? ?/sec 1.06 4.7±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.5±0.17ms ? ?/sec 1.22 44.7±0.23ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.9±0.08ms ? ?/sec 1.10 14.2±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 15.2±0.80ms ? ?/sec 1.02 15.6±0.76ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 14.7±1.09ms ? ?/sec 1.06 15.6±0.74ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 47.8±0.30ms ? ?/sec 1.15 54.9±0.33ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.8±0.08ms ? ?/sec 1.08 15.9±0.09ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.02 13.9±0.50ms ? ?/sec 1.00 13.6±0.78ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 11.4±0.78ms ? ?/sec 1.17 13.3±0.47ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.84 13.4±0.21ms ? ?/sec 1.00 4.7±0.03ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 3.30 2.9±0.01ms ? ?/sec 1.00 881.1±2.75µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 3.38 2.7±0.01ms ? ?/sec 1.00 800.7±1.29µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.70 821.9±1.55µs ? ?/sec 1.00 483.1±0.85µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.32 19.9±0.12ms ? ?/sec 1.00 15.1±0.07ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.84 4.1±0.02ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.39 3.0±0.01ms ? ?/sec 1.00 1240.2±4.74µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.30 1078.4±2.78µs ? ?/sec 1.00 828.2±1.12µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1bvfq8zp-9d43b4b4-sjl2
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6175113536-3397-68wbk 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1385.3s |
| Peak memory | 59.9 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1358.8s |
| CPU sys | 23.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1400.3s |
| Peak memory | 60.1 MiB |
| Avg memory | 19.9 MiB |
| CPU user | 1371.5s |
| CPU sys | 23.1s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175113536:finish:51b9e42628fac8e3746374a6b53d29b3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176773181
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175800199)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (4ebe938906728c7dc823c3444f55c5a0ad4a32e9) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4ebe938906728c7dc823c3444f55c5a0ad4a32e9)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.85 13.6±0.14ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.62 3.3±0.02ms ? ?/sec 1.00 917.8±3.85µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.52 2.7±0.01ms ? ?/sec 1.00 759.7±1.19µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.53 1959.3±23.15µs ? ?/sec 1.00 432.7±1.84µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.33 20.4±0.05ms ? ?/sec 1.00 15.4±0.08ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.74 4.5±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.45 3.0±0.03ms ? ?/sec 1.00 1238.5±1.69µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.11 853.6±1.76µs ? ?/sec 1.00 768.1±1.78µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.42 12.7±0.08ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.77 2.3±0.02ms ? ?/sec 1.00 839.0±3.92µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.16 1625.7±3.99µs ? ?/sec 1.00 752.6±1.00µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.54 1097.2±5.16µs ? ?/sec 1.00 431.5±1.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.33 19.5±0.04ms ? ?/sec 1.00 14.6±0.08ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.43 3.6±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.59 1972.4±5.40µs ? ?/sec 1.00 1240.0±2.68µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.10 857.2±1.74µs ? ?/sec 1.00 778.6±2.31µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.07 101.2±1.75ms ? ?/sec 1.00 94.4±1.39ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.10 5.5±0.02ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.03 3.7±0.03ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.01 2.1±0.06ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.11 108.7±1.69ms ? ?/sec 1.00 97.6±1.41ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.02 5.8±0.02ms ? ?/sec 1.00 5.7±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.03 3.7±0.03ms ? ?/sec 1.00 3.6±0.04ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.1±0.02ms ? ?/sec 1.01 2.1±0.05ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 7.73 154.3±3.16ms ? ?/sec 1.00 20.0±0.75ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.17 22.6±0.08ms ? ?/sec 1.00 19.4±0.08ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.09 6.6±0.05ms ? ?/sec 1.00 6.0±0.04ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.02 5.2±0.11ms ? ?/sec 1.00 5.1±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.5±0.13ms ? ?/sec 1.02 4.6±0.16ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.6±0.12ms ? ?/sec 1.00 32.6±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.0±0.04ms ? ?/sec 1.00 8.0±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.06 5.6±0.35ms ? ?/sec 1.00 5.3±0.11ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.0±0.07ms ? ?/sec 1.03 4.2±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.79 13.3±0.08ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.51 3.3±0.01ms ? ?/sec 1.00 928.7±4.29µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.44 2.6±0.02ms ? ?/sec 1.00 748.3±1.08µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.41 1912.5±11.02µs ? ?/sec 1.00 433.5±1.42µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.4±0.06ms ? ?/sec 1.00 15.2±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.71 4.5±0.03ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.34 2.9±0.03ms ? ?/sec 1.00 1242.5±2.03µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.11 849.3±2.10µs ? ?/sec 1.00 767.1±1.94µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.78 12.6±0.07ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.98 2.3±0.02ms ? ?/sec 1.00 786.3±2.64µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.84 1347.9±8.75µs ? ?/sec 1.00 733.2±1.61µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.34 579.5±2.13µs ? ?/sec 1.00 432.3±1.84µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.6±0.02ms ? ?/sec 1.00 14.5±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.41 3.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.38 1725.4±4.11µs ? ?/sec 1.00 1247.2±4.32µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.13 858.9±2.62µs ? ?/sec 1.00 761.7±1.35µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 15.16 11.3±0.11ms ? ?/sec 1.00 747.2±4.25µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.27 30.2±0.22ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.78 4.3±0.09ms ? ?/sec 1.00 1547.4±8.72µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.33 1697.2±3.92µs ? ?/sec 1.00 1277.2±3.40µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.08 836.8±1.95µs ? ?/sec 1.00 777.6±1.99µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.3±0.10ms ? ?/sec 1.00 33.4±0.21ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.12 7.0±0.02ms ? ?/sec 1.00 6.2±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.07 2.7±0.00ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1525.9±2.78µs ? ?/sec 1.08 1647.2±2.67µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.4±0.12ms ? ?/sec 1.23 28.9±0.12ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.01ms ? ?/sec 1.17 6.0±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.9±0.01ms ? ?/sec 1.04 3.0±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.04 2.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.11ms ? ?/sec 1.17 38.6±0.07ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.6±0.16ms ? ?/sec 1.10 7.3±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.03 3.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.01ms ? ?/sec 1.06 2.3±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.73 13.4±0.13ms ? ?/sec 1.00 4.9±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.96 3.0±0.01ms ? ?/sec 1.00 1000.3±4.04µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.79 3.2±0.01ms ? ?/sec 1.00 840.3±2.07µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.45 820.4±1.30µs ? ?/sec 1.00 565.7±1.53µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.31 20.0±0.07ms ? ?/sec 1.00 15.3±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.73 4.1±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.80 3.5±0.01ms ? ?/sec 1.00 1234.3±2.59µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.25 1069.8±3.38µs ? ?/sec 1.00 859.1±1.02µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.13ms ? ?/sec 1.35 18.5±0.20ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1555.8±8.42µs ? ?/sec 1.40 2.2±0.02ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 357.6±0.69µs ? ?/sec 1.44 514.5±2.22µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.5±0.43µs ? ?/sec 1.28 307.9±0.52µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.2±0.09ms ? ?/sec 1.27 25.8±0.24ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.27 3.2±0.02ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 617.2±1.68µs ? ?/sec 1.28 791.8±3.05µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 430.7±0.32µs ? ?/sec 1.18 506.3±0.77µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.7±0.23ms ? ?/sec 1.37 17.4±0.14ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1716.4±17.88µs ? ?/sec 1.47 2.5±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 730.3±0.98µs ? ?/sec 1.16 845.5±2.63µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 469.8±1.14µs ? ?/sec 1.21 570.1±2.10µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.1±0.05ms ? ?/sec 1.27 24.2±0.20ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.01ms ? ?/sec 1.13 3.5±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1038.5±1.49µs ? ?/sec 1.14 1183.8±2.66µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 694.3±1.27µs ? ?/sec 1.17 813.1±1.36µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.06ms ? ?/sec 1.38 17.9±0.11ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.35 3.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 1992.4±2.58µs ? ?/sec 1.06 2.1±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1480.6±4.34µs ? ?/sec 1.08 1592.8±5.71µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.7±0.12ms ? ?/sec 1.22 25.4±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.01ms ? ?/sec 1.17 4.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.00ms ? ?/sec 1.05 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1716.6±4.19µs ? ?/sec 1.07 1830.5±6.98µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.7±0.15ms ? ?/sec 1.11 38.6±0.12ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.02ms ? ?/sec 1.10 5.5±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1984.7±3.24µs ? ?/sec 1.03 2.0±0.00ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1224.8±2.58µs ? ?/sec 1.07 1307.2±3.17µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.1±0.08ms ? ?/sec 1.09 63.1±0.13ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.02ms ? ?/sec 1.06 11.5±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.3±0.01ms ? ?/sec 1.02 5.5±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.04 4.7±0.00ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.4±0.16ms ? ?/sec 1.11 49.0±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.5±0.16ms ? ?/sec 1.02 9.7±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.02ms ? ?/sec 1.01 5.3±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.03 3.8±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.0±0.16ms ? ?/sec 1.09 56.5±0.17ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.6±0.04ms ? ?/sec 1.03 10.9±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.1±0.02ms ? ?/sec 1.01 6.1±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.03ms ? ?/sec 1.03 4.5±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.3±0.24ms ? ?/sec 1.23 44.6±0.11ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.5±0.12ms ? ?/sec 1.07 13.4±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.4±0.17ms ? ?/sec 1.03 11.8±0.18ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.8±0.33ms ? ?/sec 1.00 10.9±0.27ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 47.2±0.23ms ? ?/sec 1.14 53.6±0.17ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.0±0.09ms ? ?/sec 1.06 14.9±0.09ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.04 11.1±0.33ms ? ?/sec 1.00 10.7±0.16ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 9.1±0.26ms ? ?/sec 1.15 10.4±0.54ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.70 13.1±0.12ms ? ?/sec 1.00 4.9±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.87 2.9±0.01ms ? ?/sec 1.00 1004.5±5.00µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 3.12 2.6±0.00ms ? ?/sec 1.00 841.3±1.93µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.44 817.4±1.82µs ? ?/sec 1.00 566.5±1.11µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.31 19.9±0.06ms ? ?/sec 1.00 15.2±0.06ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.68 4.1±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.28 2.8±0.00ms ? ?/sec 1.00 1228.3±2.17µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.23 1062.7±2.20µs ? ?/sec 1.00 861.9±1.50µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1r5gco5q-658e2293-nfvc
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6175800199-3403-76rc7 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1410.3s |
| Peak memory | 60.4 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1386.4s |
| CPU sys | 21.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.0 MiB |
| Avg memory | 19.9 MiB |
| CPU user | 1367.5s |
| CPU sys | 26.3s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175800199:finish:4d865e9c3483a567e63fcf837968c98b -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5782203331
@Dandandan Could you take a look? speceifally just at the API
```rust
pub fn push_batch_interleaved(
&mut self,
batches: &[&RecordBatch],
indices: &[(usize, usize)],
)
```
for datafusion at least is this the right shape? should we include a possible filter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321680067
##########
arrow-select/src/coalesce.rs:
##########
@@ -2893,4 +3048,147 @@ mod tests {
"draining must release the accounted memory"
);
}
+
+ #[test]
+ fn test_push_batch_with_indices_splits_at_target_batch_size() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..1000i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from_iter_values((0..1000u32).step_by(3));
+ let target_batch_size = 100;
+
+ let mut coalescer = BatchCoalescer::new(Arc::clone(&schema), target_batch_size);
+ coalescer
+ .push_batch_with_indices(batch.clone(), &indices)
+ .unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ let mut completed = vec![];
+ while let Some(batch) = coalescer.next_completed_batch() {
+ completed.push(batch);
+ }
+ for batch in &completed[..completed.len() - 1] {
+ assert_eq!(batch.num_rows(), target_batch_size);
+ }
+
+ let actual = concat_batches(&schema, completed.iter()).unwrap();
+ let expected = take_record_batch(&batch, &indices).unwrap();
+ assert_eq!(expected, actual);
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_empty_is_noop() {
+ let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)]));
+ let array = Int32Array::from_iter_values(0..4i32);
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(array)]).unwrap();
+ let indices = UInt32Array::from(Vec::<u32>::new());
+
+ let mut coalescer = BatchCoalescer::new(schema, 4096);
+ coalescer.push_batch_with_indices(batch, &indices).unwrap();
+ coalescer.finish_buffered_batch().unwrap();
+ assert!(coalescer.next_completed_batch().is_none());
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_schema_mismatch() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Int32, false),
+ ]));
+ let mut coalescer = BatchCoalescer::new(schema, 100);
+ let batch = RecordBatch::try_new(
+ Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, false)])),
+ vec![Arc::new(Int32Array::from_iter_values(0..5i32))],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1]);
+ let err = coalescer
+ .push_batch_with_indices(batch, &indices)
+ .unwrap_err()
+ .to_string();
+ assert!(
+ err.contains("Batch has 1 columns but BatchCoalescer expects 2"),
+ "unexpected error: {err}"
+ );
+ }
+
+ #[test]
+ fn test_push_batch_with_indices_rejects_oob_before_mutation() {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("c0", DataType::Int32, false),
+ Field::new("c1", DataType::Utf8View, false),
+ ]));
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int32Array::from_iter_values(0..16i32)),
+ Arc::new(StringViewArray::from(vec!["a"; 16])),
+ ],
+ )
+ .unwrap();
+ let indices = UInt32Array::from(vec![0u32, 1, 2, 3, 0, 1, 2, 99]);
Review Comment:
added in latest commit, `test_push_batch_with_indices_rejects_oob_in_direct_copy_chunk` + `test_push_batch_with_index_direct_rejects_oob_without_partial_write`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6293073916
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292059412)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (3946b865b87fa51a8f28629b41fa266f0cf2b176) to 1357f25 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1357f25f26ca67634e992bdb35220b4ee59c780f..3946b865b87fa51a8f28629b41fa266f0cf2b176)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.14 13.5±0.11ms ? ?/sec 1.00 3.3±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.59 3.5±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.12 3.1±0.15ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1978.4±6.89µs ? ?/sec 1.10 2.2±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 2.32 20.4±0.03ms ? ?/sec 1.00 8.8±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.47 4.5±0.01ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.3±0.14ms ? ?/sec 1.07 3.6±0.26ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 868.8±4.41µs ? ?/sec 1.00 870.6±1.65µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.82 12.7±0.09ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.04 2.3±0.01ms ? ?/sec 1.00 1141.9±2.67µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.01 1652.8±3.93µs ? ?/sec 1.00 1642.0±8.75µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1104.2±3.18µs ? ?/sec 1.01 1116.4±4.93µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 2.41 19.5±0.04ms ? ?/sec 1.00 8.1±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.81 3.5±0.01ms ? ?/sec 1.00 1949.7±6.08µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.02 2.0±0.01ms ? ?/sec 1.00 1982.5±7.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 869.3±5.69µs ? ?/sec 1.00 868.1±1.88µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.10 118.6±2.21ms ? ?/sec 1.00 108.2±3.52ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.15 5.8±0.02ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.01 3.7±0.02ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 99.7±1.38ms ? ?/sec 1.08 107.3±2.54ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.12 5.8±0.01ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.06 3.8±0.03ms ? ?/sec 1.00 3.6±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.0±0.02ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 165.5±4.42ms ? ?/sec 1.00 164.7±4.62ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.52 23.1±0.14ms ? ?/sec 1.00 15.1±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.21 6.9±0.01ms ? ?/sec 1.00 5.7±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.04ms ? ?/sec 1.23 6.4±0.34ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.5±0.07ms ? ?/sec 1.28 5.7±0.23ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.32 32.6±0.09ms ? ?/sec 1.00 24.7±0.13ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.13 8.1±0.02ms ? ?/sec 1.00 7.2±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.09 5.9±0.31ms ? ?/sec 1.00 5.4±0.18ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.10 5.8±0.27ms ? ?/sec 1.00 5.2±0.37ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.25 13.4±0.09ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.54 3.4±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.08ms ? ?/sec 1.13 3.1±0.11ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1936.4±3.43µs ? ?/sec 1.11 2.2±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 2.32 20.4±0.03ms ? ?/sec 1.00 8.8±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.57 4.4±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.2±0.20ms ? ?/sec 1.08 3.5±0.15ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 858.9±2.04µs ? ?/sec 1.01 866.8±1.58µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.90 12.6±0.08ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.11 2.3±0.02ms ? ?/sec 1.00 1086.4±3.57µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.07 1459.8±3.16µs ? ?/sec 1.00 1366.8±5.95µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 583.1±2.48µs ? ?/sec 1.00 585.7±0.89µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 2.41 19.5±0.02ms ? ?/sec 1.00 8.1±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.83 3.5±0.01ms ? ?/sec 1.00 1887.0±6.10µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.01 1753.4±3.72µs ? ?/sec 1.00 1736.8±5.36µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 862.9±3.09µs ? ?/sec 1.00 865.7±1.73µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.7±0.26ms ? ?/sec 1.01 11.8±0.21ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.18 29.9±0.24ms ? ?/sec 1.00 4.8±0.03ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.24 4.8±0.02ms ? ?/sec 1.00 1467.1±6.84µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.50 1851.9±6.36µs ? ?/sec 1.00 1232.7±1.86µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 837.7±1.01µs ? ?/sec 1.03 861.1±0.81µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 2.57 47.4±0.13ms ? ?/sec 1.00 18.5±0.09ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 2.01 6.9±0.02ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.30 2.7±0.00ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1530.5±1.79µs ? ?/sec 1.02 1562.4±2.07µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.08 23.6±0.12ms ? ?/sec 1.00 21.8±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.03 5.2±0.01ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.02 2.9±0.00ms ? ?/sec 1.00 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.03 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.04 33.1±0.09ms ? ?/sec 1.00 31.7±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.5±0.18ms ? ?/sec 1.03 6.7±0.05ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.02 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.12 13.4±0.19ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.46 3.0±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.04 3.4±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 817.2±3.07µs ? ?/sec 1.06 867.5±1.05µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 2.26 20.1±0.16ms ? ?/sec 1.00 8.9±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.55 4.1±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1065.7±1.93µs ? ?/sec 1.02 1091.8±1.14µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.07 13.7±0.27ms ? ?/sec 1.00 12.7±0.10ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.05 1572.3±25.08µs ? ?/sec 1.00 1494.0±9.57µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 359.8±2.82µs ? ?/sec 1.10 397.3±0.90µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 239.8±0.32µs ? ?/sec 1.15 275.5±0.26µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.06 20.4±0.26ms ? ?/sec 1.00 19.3±0.12ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.03 2.5±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 621.9±3.26µs ? ?/sec 1.06 658.5±0.87µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 433.7±0.43µs ? ?/sec 1.08 468.3±0.51µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.15 12.4±0.19ms ? ?/sec 1.00 10.8±0.09ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.11 1785.5±10.66µs ? ?/sec 1.00 1603.9±10.84µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 745.8±1.61µs ? ?/sec 1.01 755.6±2.60µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 462.7±0.73µs ? ?/sec 1.07 496.0±0.80µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.09 18.9±0.09ms ? ?/sec 1.00 17.4±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.03 3.1±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1060.1±1.77µs ? ?/sec 1.03 1090.3±2.41µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 715.9±1.48µs ? ?/sec 1.02 732.1±3.38µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.18 13.2±0.15ms ? ?/sec 1.00 11.1±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.09 2.5±0.01ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.02 2.1±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1523.4±2.73µs ? ?/sec 1.00 1525.7±3.22µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.09 20.3±0.07ms ? ?/sec 1.00 18.6±0.08ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.04 3.7±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1759.6±5.56µs ? ?/sec 1.01 1776.6±13.56µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.04 34.7±0.16ms ? ?/sec 1.00 33.5±0.13ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.01 5.0±0.01ms ? ?/sec 1.00 5.0±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.05 2.0±0.00ms ? ?/sec 1.00 1914.2±3.09µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1233.9±1.59µs ? ?/sec 1.02 1257.0±1.59µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.01 58.2±0.22ms ? ?/sec 1.00 57.3±0.15ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.5±0.01ms ? ?/sec 1.00 10.5±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 4.9±0.01ms ? ?/sec 1.00 4.9±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.01ms ? ?/sec 1.01 4.3±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.05 44.8±0.19ms ? ?/sec 1.00 42.8±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.02 9.5±0.24ms ? ?/sec 1.00 9.2±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.3±0.02ms ? ?/sec 1.01 5.3±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.02ms ? ?/sec 1.01 3.7±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.02 51.5±0.19ms ? ?/sec 1.00 50.4±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.03 10.5±0.05ms ? ?/sec 1.00 10.2±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.1±0.02ms ? ?/sec 1.01 6.2±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.03ms ? ?/sec 1.02 4.5±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.05 36.9±0.26ms ? ?/sec 1.00 35.3±0.13ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.05 13.2±0.03ms ? ?/sec 1.00 12.6±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.5±0.08ms ? ?/sec 1.25 14.5±1.02ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.4±0.10ms ? ?/sec 1.34 14.0±0.53ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.03 47.1±0.17ms ? ?/sec 1.00 45.8±0.32ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.0±0.05ms ? ?/sec 1.00 14.0±0.06ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 11.9±0.60ms ? ?/sec 1.00 11.9±0.20ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.20 13.0±0.71ms ? ?/sec 1.00 10.8±0.49ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.06 13.3±0.12ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.49 2.9±0.01ms ? ?/sec 1.00 1960.1±4.16µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.05 2.8±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 822.8±0.78µs ? ?/sec 1.05 861.3±1.97µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 2.23 19.9±0.10ms ? ?/sec 1.00 8.9±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.59 4.0±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.02 2.9±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1066.9±1.40µs ? ?/sec 1.02 1092.4±2.00µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1mf5hvbt-40d46aa4-pf4h
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6292059412-3638-9pspn 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1405.3s |
| Peak memory | 61.7 MiB |
| Avg memory | 20.6 MiB |
| CPU user | 1377.1s |
| CPU sys | 22.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 59.6 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1367.2s |
| CPU sys | 22.5s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6292059412:finish:61d4e754908c84f7d0b34531e8b4ec38 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6305286024
> The precedent I saw was InProgressArray::copy_rows. It is a safe trait method with no documented precondition, and its impls call get_unchecked on offset..offset + len behind a SAFETY comment that relies on the caller ([primitive.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/primitive.rs#L167-L169), [byte_view.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/byte_view.rs#L411-L412)).
+1 on this. Im not sure why we need to mark the entire method as unsafe if we check for invariants before calling it. since the method is private, users cannot directly shoot themselves in the foot with this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4329746155
##########
arrow-select/src/coalesce.rs:
##########
@@ -336,11 +404,81 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ if let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
why only validate u32s?
Also this seems like it validates the indices, and then calls take_record_batch which validates the indices again
##########
arrow-select/src/coalesce.rs:
##########
@@ -312,15 +377,18 @@ impl BatchCoalescer {
/// This is semantically equivalent of calling [`Self::push_batch`]
/// with the results from [`take_record_batch`]
///
+ /// Returns `Err(ArrowError::InvalidArgumentError)` if `indices` is a
+ /// non-null `UInt32Array` with any out-of-bounds value.
+ ///
/// # Example
/// ```
- /// # use arrow_array::{record_batch, UInt64Array};
+ /// # use arrow_array::{record_batch, UInt32Array};
Review Comment:
does this mean we can't use UInt64 as indexes anymore? According to Codex they would stil work but not get the new fast path
##########
arrow-select/src/coalesce/byte_view.rs:
##########
@@ -354,13 +362,156 @@ impl<B: ByteViewType> InProgressByteViewArray<B> {
self.views.extend(new_views);
self.current = Some(dst_buffer);
}
+
+ #[inline(never)]
+ fn append_indexed_views_and_copy_strings(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ ) {
+ self.append_indexed_views_and_copy_strings_with_max_buffer_size(
+ src_views,
+ indices,
+ src_buffers,
+ selected_bytes,
+ i32::MAX as usize,
+ );
+ }
+
+ fn append_indexed_views_and_copy_strings_with_max_buffer_size(
+ &mut self,
+ src_views: &[u128],
+ indices: &[u32],
+ src_buffers: &[Buffer],
+ selected_bytes: usize,
+ max_buffer_size: usize,
+ ) {
+ debug_assert!(
+ indices.iter().all(|&idx| (idx as usize) < src_views.len()),
+ "append_indexed_views_and_copy_strings_with_max_buffer_size requires caller-validated in-bounds indices",
+ );
+ if selected_bytes == 0 {
+ self.views.reserve(indices.len());
+ for &idx in indices {
+ // SAFETY: caller validated idx < src_views.len().
+ let raw = unsafe { *src_views.get_unchecked(idx as usize) };
Review Comment:
I don't think the caller was required to do any validation (the comments say they have to)
`debug_assert` isn't sufficient to ensure this invariant in release builds -- we either need to actually verify it or mark the trait function as `unsafe`
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
Review Comment:
Update; see my comment above -- I think this actually needs to be marked `unsafe`
##########
arrow-select/src/coalesce.rs:
##########
@@ -336,11 +404,81 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ if let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
+ }
+
+ let exceeds_byte_view_threshold = self.has_byte_view_columns
+ && !should_use_byte_view_indexed_copy(batch.num_rows(), indices.len());
+ if !should_use_indexed_copy(batch.num_rows(), indices.len()) || exceeds_byte_view_threshold
+ {
+ let taken_batch = take_record_batch(&batch, indices)?;
+ return self.push_batch(taken_batch);
+ }
+
+ let fast_path_allowed = self
+ .biggest_coalesce_batch_size
+ .is_none_or(|limit| indices.len() <= limit);
+ if fast_path_allowed
+ && let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ return self.push_batch_with_index(batch, u32_arr);
+ }
let taken_batch = take_record_batch(&batch, indices)?;
self.push_batch(taken_batch)
}
+ fn push_batch_with_index(
Review Comment:
could we commet that this is a fast path and under what circumstance sit is allowed?
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
Review Comment:
I found this somewhat confusing as it says ``indices` must already be bounds-checked against `source`.` but doesn't say what happens if this is not the case (I was wondering if it could lead to out of bounds erorrs)
However, after review I think the result is that it panics
So I recommend adding a note here to clarify:
```suggestion
/// # Panics
/// Pancis if any value in `indices` is not a valid index in `source`.
##########
arrow-select/src/coalesce.rs:
##########
@@ -41,6 +42,49 @@ use fixed_size_binary::InProgressFixedSizeBinaryArray;
use generic::GenericInProgressArray;
use primitive::InProgressPrimitiveArray;
+// Caller must have bounds-checked `indices` against `source_nulls`.
Review Comment:
can we please document what happens if this is not the case?
If it results in an out of bounds access we need to mark this function as `unsafe` -- a debug assert is not enough to ensure this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6223309703
Working on this at the moment. The issue is that to the size of the indices isn't a very useful hint. We'd need to traverse the entire indices array once & then make a decision but this might drag down the performance for low selectivity cases.
The goal of this new method is to replace the take + push batch, that repartitionExex does so the low selectivity speed ups should be prioritized. Since data is typically 1/8 or 1/16 or 1/32.... for output partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6223245706
Classic database tradeoff!
I wonder if you can find some way to pick / detect the two cases and call the special code for each of the cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216906
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4202721363
##########
arrow-select/src/coalesce.rs:
##########
@@ -2893,4 +3084,5 @@ mod tests {
"draining must release the accounted memory"
);
}
+
Review Comment:
remove
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +974,32 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at positions `indices` from `source` into the in-progress
+ /// array. Fallback path: builds a `UInt32Array` from `indices` without
+ /// cloning the slice, then calls [`take`]. Types with a fast path
+ /// (primitive, byte-view) override this to scatter directly from `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ // Avoid cloning `indices` into a fresh Vec: wrap the existing slice in
+ // a `ScalarBuffer` (zero-copy via `Buffer::from_slice_ref` when the
+ // slice outlives the kernel call, which it does here).
+ use arrow_buffer::{Buffer, ScalarBuffer};
+ let buf = Buffer::from_slice_ref(indices);
Review Comment:
remove
##########
arrow-select/src/coalesce.rs:
##########
@@ -721,6 +723,169 @@ impl BatchCoalescer {
}
}
+/// Row buffer used by [`PartitionCoalescer`].
+#[derive(Debug)]
+struct BufferCoalescer {
+ schema: SchemaRef,
+ target_batch_size: usize,
+ in_progress_arrays: Vec<Box<dyn InProgressArray>>,
+ buffered_rows: usize,
+ completed: VecDeque<RecordBatch>,
+}
+
+impl BufferCoalescer {
+ fn new(schema: SchemaRef, target_batch_size: usize) -> Self {
+ let in_progress_arrays = schema
+ .fields()
+ .iter()
+ .map(|field| create_in_progress_array(field.data_type(), target_batch_size))
+ .collect();
+ Self {
+ schema,
+ target_batch_size,
+ in_progress_arrays,
+ buffered_rows: 0,
+ completed: VecDeque::with_capacity(1),
+ }
+ }
+
+ fn push_rows(
+ &mut self,
+ batch: &RecordBatch,
+ rows: &[u32],
+ ) -> Result<(), ArrowError> {
+ if rows.is_empty() {
+ return Ok(());
+ }
+ let columns = batch.columns();
+ if columns.len() != self.in_progress_arrays.len() {
+ return Err(ArrowError::InvalidArgumentError(format!(
+ "Batch has {} columns but BufferCoalescer expects {}",
+ columns.len(),
+ self.in_progress_arrays.len()
+ )));
+ }
+
+ let mut row_offset = 0;
+ let mut remaining = rows.len();
+ while remaining > self.target_batch_size - self.buffered_rows {
+ let room = self.target_batch_size - self.buffered_rows;
+ let chunk = &rows[row_offset..row_offset + room];
+ for (in_progress, column) in self.in_progress_arrays.iter_mut().zip(columns) {
+ in_progress.copy_rows_by_indices(column, chunk)?;
+ }
+ self.buffered_rows += room;
+ row_offset += room;
+ remaining -= room;
+ self.flush()?;
+ }
+ if remaining > 0 {
+ let chunk = &rows[row_offset..row_offset + remaining];
+ for (in_progress, column) in self.in_progress_arrays.iter_mut().zip(columns) {
+ in_progress.copy_rows_by_indices(column, chunk)?;
+ }
+ self.buffered_rows += remaining;
+ }
+ Ok(())
+ }
+
+ fn next_completed_batch(&mut self) -> Option<RecordBatch> {
+ self.completed.pop_front()
+ }
+
+ fn finish(&mut self) -> Result<(), ArrowError> {
+ self.flush()
+ }
+
+ fn flush(&mut self) -> Result<(), ArrowError> {
+ if self.buffered_rows == 0 {
+ return Ok(());
+ }
+ let arrays = self
+ .in_progress_arrays
+ .iter_mut()
+ .map(|in_progress| in_progress.finish())
+ .collect::<Result<Vec<_>, ArrowError>>()?;
+ let batch = unsafe {
+ RecordBatch::new_unchecked(Arc::clone(&self.schema), arrays, self.buffered_rows)
+ };
+ self.buffered_rows = 0;
+ self.completed.push_back(batch);
+ Ok(())
+ }
+}
+
+/// Single-partition coalescer: filters an input batch down to rows whose
+/// partition-id equals `my_partition`, buffering them into target-sized
+/// batches. Each output partition in a repartition operator owns one of these.
+#[derive(Debug)]
+pub struct PartitionCoalescer {
+ my_partition: u64,
+ /// Reused scratch of row indices where `pid == my_partition`.
+ bucket: Vec<u32>,
+ coalescer: BufferCoalescer,
+}
+
+impl PartitionCoalescer {
+ /// Create a coalescer that buffers rows tagged with `my_partition`.
+ pub fn new(schema: SchemaRef, target_batch_size: usize, my_partition: usize) -> Self {
+ Self {
+ my_partition: my_partition as u64,
+ bucket: Vec::new(),
+ coalescer: BufferCoalescer::new(schema, target_batch_size),
+ }
+ }
+
+ /// Append rows from `batch` whose partition-id equals `my_partition`.
+ pub fn push(
+ &mut self,
+ batch: &RecordBatch,
+ partition_ids: &UInt64Array,
+ ) -> Result<(), ArrowError> {
+ let pids = partition_ids.values();
+ self.bucket.clear();
+ self.bucket.reserve(pids.len());
+ let my = self.my_partition;
+ unsafe {
+ let base = self.bucket.as_mut_ptr();
+ let mut out = base;
+ // Branchless conditional store.
+ for (row, &pid) in pids.iter().enumerate() {
+ *out = row as u32;
+ out = out.add((pid == my) as usize);
+ }
+ self.bucket.set_len(out.offset_from(base) as usize);
+ }
+ if !self.bucket.is_empty() {
+ self.coalescer.push_rows(batch, &self.bucket)?;
+ }
+ Ok(())
+ }
+
+ /// Append a pre-bucketed slice of row indices.
Review Comment:
better doc comment here
##########
arrow-select/src/coalesce.rs:
##########
@@ -721,6 +723,169 @@ impl BatchCoalescer {
}
}
+/// Row buffer used by [`PartitionCoalescer`].
+#[derive(Debug)]
+struct BufferCoalescer {
+ schema: SchemaRef,
+ target_batch_size: usize,
+ in_progress_arrays: Vec<Box<dyn InProgressArray>>,
+ buffered_rows: usize,
+ completed: VecDeque<RecordBatch>,
+}
+
+impl BufferCoalescer {
+ fn new(schema: SchemaRef, target_batch_size: usize) -> Self {
+ let in_progress_arrays = schema
+ .fields()
+ .iter()
+ .map(|field| create_in_progress_array(field.data_type(), target_batch_size))
+ .collect();
+ Self {
+ schema,
+ target_batch_size,
+ in_progress_arrays,
+ buffered_rows: 0,
+ completed: VecDeque::with_capacity(1),
+ }
+ }
+
+ fn push_rows(
+ &mut self,
+ batch: &RecordBatch,
+ rows: &[u32],
+ ) -> Result<(), ArrowError> {
+ if rows.is_empty() {
+ return Ok(());
+ }
+ let columns = batch.columns();
+ if columns.len() != self.in_progress_arrays.len() {
+ return Err(ArrowError::InvalidArgumentError(format!(
+ "Batch has {} columns but BufferCoalescer expects {}",
+ columns.len(),
+ self.in_progress_arrays.len()
+ )));
+ }
+
+ let mut row_offset = 0;
+ let mut remaining = rows.len();
+ while remaining > self.target_batch_size - self.buffered_rows {
+ let room = self.target_batch_size - self.buffered_rows;
+ let chunk = &rows[row_offset..row_offset + room];
+ for (in_progress, column) in self.in_progress_arrays.iter_mut().zip(columns) {
+ in_progress.copy_rows_by_indices(column, chunk)?;
+ }
+ self.buffered_rows += room;
+ row_offset += room;
+ remaining -= room;
+ self.flush()?;
+ }
+ if remaining > 0 {
+ let chunk = &rows[row_offset..row_offset + remaining];
+ for (in_progress, column) in self.in_progress_arrays.iter_mut().zip(columns) {
+ in_progress.copy_rows_by_indices(column, chunk)?;
+ }
+ self.buffered_rows += remaining;
+ }
+ Ok(())
+ }
+
+ fn next_completed_batch(&mut self) -> Option<RecordBatch> {
+ self.completed.pop_front()
+ }
+
+ fn finish(&mut self) -> Result<(), ArrowError> {
+ self.flush()
+ }
+
+ fn flush(&mut self) -> Result<(), ArrowError> {
+ if self.buffered_rows == 0 {
+ return Ok(());
+ }
+ let arrays = self
+ .in_progress_arrays
+ .iter_mut()
+ .map(|in_progress| in_progress.finish())
+ .collect::<Result<Vec<_>, ArrowError>>()?;
+ let batch = unsafe {
+ RecordBatch::new_unchecked(Arc::clone(&self.schema), arrays, self.buffered_rows)
+ };
+ self.buffered_rows = 0;
+ self.completed.push_back(batch);
+ Ok(())
+ }
+}
+
+/// Single-partition coalescer: filters an input batch down to rows whose
+/// partition-id equals `my_partition`, buffering them into target-sized
+/// batches. Each output partition in a repartition operator owns one of these.
+#[derive(Debug)]
+pub struct PartitionCoalescer {
+ my_partition: u64,
+ /// Reused scratch of row indices where `pid == my_partition`.
+ bucket: Vec<u32>,
+ coalescer: BufferCoalescer,
+}
+
+impl PartitionCoalescer {
+ /// Create a coalescer that buffers rows tagged with `my_partition`.
+ pub fn new(schema: SchemaRef, target_batch_size: usize, my_partition: usize) -> Self {
+ Self {
+ my_partition: my_partition as u64,
+ bucket: Vec::new(),
+ coalescer: BufferCoalescer::new(schema, target_batch_size),
+ }
+ }
+
+ /// Append rows from `batch` whose partition-id equals `my_partition`.
Review Comment:
better doc comment here. maybe an example would nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5872526624
@alamb @Dandandan bumping this as its very related to https://github.com/apache/datafusion/issues/7001#issuecomment-5801503499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved + trait methods [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-5983449673
I think im going to revamp this PR. I dont like the current API thats being used & I dont like the scope creep.
Ill explain more in the refrenced issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274628165
> @alamb, @mbutrovich this is ready for review, I cant mark it as ready for review since I have 3 other PR's open 😅
Which one should I review first? i'll try and get one in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6275326509
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274351136)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (41bbe160f8e526fb621f0e2de3ebdead08cf45d2) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..41bbe160f8e526fb621f0e2de3ebdead08cf45d2)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.39 13.7±0.11ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.54 3.4±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.04 3.0±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1992.1±3.83µs ? ?/sec 1.00 1998.7±3.10µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.5±0.06ms ? ?/sec 1.00 15.3±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.12 4.6±0.01ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.01 3.1±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 875.7±1.30µs ? ?/sec 1.02 893.6±1.30µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.95 12.8±0.11ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 1.98 2.3±0.01ms ? ?/sec 1.00 1180.9±8.15µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1683.4±4.08µs ? ?/sec 1.01 1696.9±4.05µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1115.4±3.98µs ? ?/sec 1.02 1133.2±6.28µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.36 19.6±0.07ms ? ?/sec 1.00 14.5±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.18 3.6±0.01ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 867.4±1.00µs ? ?/sec 1.01 877.9±0.72µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.04 101.0±1.38ms ? ?/sec 1.00 97.4±0.79ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.01 5.7±0.01ms ? ?/sec 1.00 5.6±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.9±0.03ms ? ?/sec 1.00 3.9±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.07 102.5±3.38ms ? ?/sec 1.00 96.1±0.87ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.9±0.01ms ? ?/sec 1.01 5.9±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.03 3.8±0.03ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.02 2.1±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 152.7±2.52ms ? ?/sec 1.03 156.8±1.91ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 22.6±0.10ms ? ?/sec 1.02 23.1±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 6.7±0.02ms ? ?/sec 1.01 6.8±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.7±0.09ms ? ?/sec 1.03 5.9±0.09ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.01 4.4±0.05ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.02 33.4±0.18ms ? ?/sec 1.00 32.7±0.10ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.02 8.5±0.02ms ? ?/sec 1.00 8.3±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.2±0.03ms ? ?/sec 1.01 5.3±0.05ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.0±0.06ms ? ?/sec 1.03 4.2±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.39 13.5±0.07ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.59 3.4±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.01ms ? ?/sec 1.04 2.8±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 1968.9±4.22µs ? ?/sec 1.01 1991.7±8.96µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.4±0.04ms ? ?/sec 1.00 15.2±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.17 4.5±0.01ms ? ?/sec 1.00 3.9±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.01ms ? ?/sec 1.01 3.0±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 864.7±1.56µs ? ?/sec 1.02 881.2±1.06µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.03 12.7±0.06ms ? ?/sec 1.00 2.5±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.12 2.3±0.01ms ? ?/sec 1.00 1091.5±9.64µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.00 1396.2±2.97µs ? ?/sec 1.02 1421.1±4.22µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 588.7±0.73µs ? ?/sec 1.02 598.8±1.16µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.37 19.5±0.05ms ? ?/sec 1.00 14.3±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.19 3.5±0.01ms ? ?/sec 1.00 2.9±0.07ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1786.5±3.40µs ? ?/sec 1.00 1789.8±16.94µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 868.4±1.09µs ? ?/sec 1.01 878.6±1.62µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.07 12.3±0.11ms ? ?/sec 1.00 11.5±0.05ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.29 30.5±0.28ms ? ?/sec 1.00 4.8±0.03ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.16 4.7±0.05ms ? ?/sec 1.00 1487.6±26.61µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.35 1727.7±3.66µs ? ?/sec 1.00 1280.1±2.91µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 839.5±0.94µs ? ?/sec 1.00 835.4±0.93µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.41 47.4±0.24ms ? ?/sec 1.00 33.5±0.15ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.12 7.0±0.03ms ? ?/sec 1.00 6.2±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.07 2.7±0.01ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1553.5±2.07µs ? ?/sec 1.00 1546.6±1.57µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.01 23.7±0.17ms ? ?/sec 1.00 23.5±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.08 5.3±0.02ms ? ?/sec 1.00 4.9±0.16ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.9±0.00ms ? ?/sec 1.02 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.00ms ? ?/sec 1.00 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.01 33.6±0.22ms ? ?/sec 1.00 33.4±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.02 6.7±0.12ms ? ?/sec 1.00 6.6±0.17ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.03ms ? ?/sec 1.02 3.0±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.01 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.17 13.4±0.20ms ? ?/sec 1.00 4.2±0.06ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.47 3.0±0.01ms ? ?/sec 1.00 2.0±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.3±0.00ms ? ?/sec 1.03 3.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 834.2±1.08µs ? ?/sec 1.07 889.1±5.63µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.35 20.2±0.16ms ? ?/sec 1.00 15.0±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.16 4.1±0.02ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.5±0.01ms ? ?/sec 1.02 3.6±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1081.6±2.26µs ? ?/sec 1.04 1125.6±2.83µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.6±0.09ms ? ?/sec 1.02 13.9±0.07ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1563.3±8.76µs ? ?/sec 1.01 1575.8±8.92µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 361.8±0.54µs ? ?/sec 1.00 363.5±0.92µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.02 244.1±0.22µs ? ?/sec 1.00 240.1±0.35µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.4±0.10ms ? ?/sec 1.00 20.4±0.07ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 618.7±1.20µs ? ?/sec 1.01 622.8±1.31µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 434.8±0.26µs ? ?/sec 1.00 432.7±0.52µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.6±0.13ms ? ?/sec 1.00 12.6±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1695.9±12.90µs ? ?/sec 1.04 1760.5±8.86µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.02 751.1±1.22µs ? ?/sec 1.00 733.1±1.42µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 452.2±0.66µs ? ?/sec 1.04 471.1±0.65µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.5±0.08ms ? ?/sec 1.00 19.5±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.0±0.02ms ? ?/sec 1.04 3.1±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1051.9±1.34µs ? ?/sec 1.01 1059.2±1.75µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 702.9±0.68µs ? ?/sec 1.01 712.1±0.68µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.07ms ? ?/sec 1.02 13.4±0.08ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.16 2.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 2.1±0.00ms ? ?/sec 1.02 2.2±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1565.3±5.95µs ? ?/sec 1.00 1567.4±2.96µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.7±0.06ms ? ?/sec 1.01 20.8±0.07ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.03ms ? ?/sec 1.02 3.8±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.4±0.00ms ? ?/sec 1.01 2.5±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1788.8±2.93µs ? ?/sec 1.01 1798.7±4.41µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.9±0.28ms ? ?/sec 1.00 34.8±0.15ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.02ms ? ?/sec 1.00 5.0±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1954.0±6.01µs ? ?/sec 1.01 1981.7±3.61µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1238.5±1.47µs ? ?/sec 1.00 1241.8±1.86µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.5±0.19ms ? ?/sec 1.01 59.0±0.23ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.0±0.03ms ? ?/sec 1.00 11.0±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.2±0.01ms ? ?/sec 1.02 5.3±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.01 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.2±0.12ms ? ?/sec 1.01 44.8±0.23ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.7±0.03ms ? ?/sec 1.01 9.7±0.07ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.4±0.01ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.5±0.12ms ? ?/sec 1.01 52.2±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.6±0.04ms ? ?/sec 1.04 11.0±0.19ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.2±0.02ms ? ?/sec 1.01 6.2±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.03ms ? ?/sec 1.00 4.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.0±0.17ms ? ?/sec 1.03 37.0±0.12ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.8±0.05ms ? ?/sec 1.02 13.1±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.9±0.13ms ? ?/sec 1.06 12.6±0.69ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.03 10.7±0.20ms ? ?/sec 1.00 10.3±0.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.01 48.0±0.24ms ? ?/sec 1.00 47.5±0.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.4±0.04ms ? ?/sec 1.00 14.4±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.8±0.07ms ? ?/sec 1.00 10.9±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 9.0±0.08ms ? ?/sec 1.02 9.2±0.15ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.10 13.1±0.14ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.51 3.0±0.01ms ? ?/sec 1.00 1952.5±15.02µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.00ms ? ?/sec 1.04 2.8±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 824.3±0.93µs ? ?/sec 1.08 890.2±1.12µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.41 20.3±0.10ms ? ?/sec 1.00 14.4±0.06ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.23 4.1±0.02ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.01ms ? ?/sec 1.02 3.0±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1061.9±1.98µs ? ?/sec 1.05 1115.1±2.54µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-14meshs7-a1510793-n2h6
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6274351136-3606-q7th2 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 60.6 MiB |
| Avg memory | 20.8 MiB |
| CPU user | 1368.9s |
| CPU sys | 20.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1375.3s |
| Peak memory | 59.1 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1348.1s |
| CPU sys | 21.6s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6274351136:finish:28799c81ccd7bece071b2e46b291c33e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292219011
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6220023990
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219043135)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a002f6344e0ec90742c2870d74c5aeed792cca9f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a002f6344e0ec90742c2870d74c5aeed792cca9f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.33 13.5±0.10ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.72 3.4±0.01ms ? ?/sec 1.00 1956.4±5.24µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 3.1±0.04ms ? ?/sec 1.01 3.1±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.1±0.02ms ? ?/sec 1.16 2.4±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.5±0.07ms ? ?/sec 1.00 15.1±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.22 4.5±0.03ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.2±0.04ms ? ?/sec 1.12 3.6±0.05ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 853.2±1.26µs ? ?/sec 1.66 1416.0±6.71µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.06 12.7±0.11ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 1.93 2.3±0.01ms ? ?/sec 1.00 1190.6±5.09µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.01 1653.4±3.28µs ? ?/sec 1.00 1633.3±8.87µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 1099.1±4.08µs ? ?/sec 1.14 1251.5±11.48µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.39 19.5±0.10ms ? ?/sec 1.00 14.1±0.06ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.18 3.5±0.01ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1953.6±5.03µs ? ?/sec 1.07 2.1±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 845.8±1.15µs ? ?/sec 1.27 1076.2±2.55µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.05 123.1±1.04ms ? ?/sec 1.00 116.9±1.23ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.04 5.6±0.01ms ? ?/sec 1.00 5.4±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.7±0.02ms ? ?/sec 1.05 3.9±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.00 109.6±0.91ms ? ?/sec 1.07 117.0±1.10ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.9±0.08ms ? ?/sec 1.00 5.9±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.7±0.02ms ? ?/sec 1.03 3.8±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.01 2.1±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.02 171.7±2.10ms ? ?/sec 1.00 169.2±1.64ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.03 23.4±0.19ms ? ?/sec 1.00 22.6±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.02 6.7±0.02ms ? ?/sec 1.00 6.6±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 6.0±0.08ms ? ?/sec 1.03 6.1±0.22ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 5.5±0.14ms ? ?/sec 1.03 5.7±0.17ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.01 32.8±0.14ms ? ?/sec 1.00 32.4±0.12ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.2±0.10ms ? ?/sec 1.02 8.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.10ms ? ?/sec 1.03 6.1±0.23ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.08 5.3±0.12ms ? ?/sec 1.00 4.9±0.10ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 4.43 13.4±0.08ms ? ?/sec 1.00 3.0±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 1.75 3.3±0.01ms ? ?/sec 1.00 1904.9±4.69µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.04ms ? ?/sec 1.02 2.9±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.02ms ? ?/sec 1.21 2.5±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.36 20.4±0.05ms ? ?/sec 1.00 15.1±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.22 4.5±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 1.00 3.1±0.05ms ? ?/sec 1.07 3.3±0.05ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.00 845.4±1.16µs ? ?/sec 1.65 1396.0±5.51µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 5.17 12.6±0.07ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.10 2.3±0.02ms ? ?/sec 1.00 1101.2±2.68µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.04 1379.7±2.77µs ? ?/sec 1.00 1325.3±2.30µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.00 573.2±1.48µs ? ?/sec 1.24 713.6±1.44µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.43 19.8±0.05ms ? ?/sec 1.00 13.8±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.20 3.5±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.00 1725.1±3.33µs ? ?/sec 1.06 1832.5±9.48µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.00 852.6±2.98µs ? ?/sec 1.37 1170.9±2.24µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.7±0.10ms ? ?/sec 1.03 12.0±0.27ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.29 30.3±0.26ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.74 4.7±0.08ms ? ?/sec 1.00 1725.2±95.43µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.36 1763.2±3.80µs ? ?/sec 1.00 1296.0±2.52µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 827.6±1.31µs ? ?/sec 1.00 826.5±1.00µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.44 47.3±0.17ms ? ?/sec 1.00 33.0±0.19ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.20 6.9±0.02ms ? ?/sec 1.00 5.7±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.04 2.6±0.00ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1532.4±2.18µs ? ?/sec 1.09 1666.1±3.19µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.4±0.09ms ? ?/sec 1.00 23.5±0.09ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.02 5.2±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.03 2.9±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.01 2.0±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.01 33.0±0.12ms ? ?/sec 1.00 32.8±0.08ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.3±0.05ms ? ?/sec 1.07 6.7±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.01 3.0±0.02ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.01 2.2±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 3.18 13.4±0.16ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 1.37 3.0±0.01ms ? ?/sec 1.00 2.2±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 1.00 3.2±0.01ms ? ?/sec 1.14 3.7±0.00ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.00 819.6±1.40µs ? ?/sec 1.96 1605.1±5.41µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.35 19.9±0.16ms ? ?/sec 1.00 14.7±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.10 4.1±0.02ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.4±0.01ms ? ?/sec 1.21 4.1±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.00 1057.6±1.21µs ? ?/sec 2.45 2.6±0.00ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.5±0.08ms ? ?/sec 1.00 13.6±0.08ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1556.8±5.86µs ? ?/sec 1.00 1559.2±5.73µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 358.2±0.60µs ? ?/sec 1.01 360.8±1.21µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.01 241.4±0.29µs ? ?/sec 1.00 238.1±0.37µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.1±0.06ms ? ?/sec 1.00 20.2±0.06ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.01 2.5±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 614.9±0.96µs ? ?/sec 1.01 621.8±1.35µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.1±0.24µs ? ?/sec 1.01 433.7±0.37µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.6±0.08ms ? ?/sec 1.03 13.0±0.05ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1713.0±12.28µs ? ?/sec 1.03 1766.1±5.39µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.01 703.1±1.84µs ? ?/sec 1.00 693.7±1.15µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.04 469.2±0.93µs ? ?/sec 1.00 449.7±1.08µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.1±0.09ms ? ?/sec 1.00 19.2±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1016.3±1.71µs ? ?/sec 1.02 1034.8±1.80µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 682.5±1.14µs ? ?/sec 1.02 695.4±1.01µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 12.9±0.06ms ? ?/sec 1.02 13.2±0.17ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.05 2.6±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.02 2.0±0.00ms ? ?/sec 1.00 2.0±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1468.0±2.29µs ? ?/sec 1.01 1487.6±2.58µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.8±0.08ms ? ?/sec 1.00 20.8±0.14ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.01ms ? ?/sec 1.01 3.8±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.3±0.00ms ? ?/sec 1.01 2.3±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1710.8±3.87µs ? ?/sec 1.01 1735.9±5.92µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.01 34.8±0.10ms ? ?/sec 1.00 34.6±0.17ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.01ms ? ?/sec 1.00 5.0±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.07 1997.9±5.90µs ? ?/sec 1.00 1866.6±5.36µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1200.0±2.54µs ? ?/sec 1.00 1201.8±8.90µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.01 58.6±0.09ms ? ?/sec 1.00 58.3±0.08ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.8±0.01ms ? ?/sec 1.01 10.9±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.2±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.3±0.14ms ? ?/sec 1.01 44.7±0.32ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.06 9.6±0.10ms ? ?/sec 1.00 9.1±0.05ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.02 5.2±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.6±0.14ms ? ?/sec 1.00 51.6±0.24ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.6±0.13ms ? ?/sec 1.01 10.7±0.04ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.02ms ? ?/sec 1.01 5.9±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.01 4.4±0.03ms ? ?/sec 1.00 4.3±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 36.1±0.14ms ? ?/sec 1.01 36.5±0.10ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.02 12.8±0.04ms ? ?/sec 1.00 12.5±0.06ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 13.6±0.30ms ? ?/sec 1.01 13.7±0.28ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.08 14.2±0.33ms ? ?/sec 1.00 13.2±0.54ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.6±0.25ms ? ?/sec 1.00 46.5±0.15ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 14.1±0.13ms ? ?/sec 1.00 14.2±0.07ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.01 12.1±0.21ms ? ?/sec 1.00 11.9±0.42ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.05 12.0±0.33ms ? ?/sec 1.00 11.5±0.24ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 3.28 13.3±0.18ms ? ?/sec 1.00 4.1±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 1.51 2.9±0.01ms ? ?/sec 1.00 1943.7±7.09µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 1.00 2.7±0.00ms ? ?/sec 1.13 3.0±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.00 819.8±0.93µs ? ?/sec 2.03 1667.9±3.82µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.41 19.9±0.09ms ? ?/sec 1.00 14.1±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.23 4.1±0.02ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.21 3.3±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.00 1049.8±2.40µs ? ?/sec 2.44 2.6±0.00ms ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1n7wuryf-80ae9fa8-7rx4
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6219043135-3510-mm8v9 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1390.3s |
| Peak memory | 60.7 MiB |
| Avg memory | 20.7 MiB |
| CPU user | 1361.8s |
| CPU sys | 23.7s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1395.3s |
| Peak memory | 61.2 MiB |
| Avg memory | 20.4 MiB |
| CPU user | 1369.0s |
| CPU sys | 23.7s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6219043135:finish:076715eb068432401b28ce10a7f832ba -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321652852
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
Review Comment:
>copy_rows_by_indices is a safe method, but the primitive and byte-view impls read through indices with unchecked pointer arithmetic. A caller that skips validate_indices_bounds gets undefined behavior rather than an error.
these traits and methods are private so users cant call this code arbitrarily. their respective conditions are outlined as safety comments for both `primitve` and `View` types. I think this method should remain safe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321810041
##########
arrow-select/src/coalesce.rs:
##########
@@ -335,11 +396,71 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ let exceeds_byte_view_threshold = self.has_byte_view_columns
+ && !should_use_byte_view_indexed_copy(batch.num_rows(), indices.len());
+ if exceeds_byte_view_threshold
+ && let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
addressed with `test_push_batch_with_indices_oob_error_is_uniform_across_schemas`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4329817979
##########
arrow-select/src/coalesce.rs:
##########
@@ -312,15 +377,18 @@ impl BatchCoalescer {
/// This is semantically equivalent of calling [`Self::push_batch`]
/// with the results from [`take_record_batch`]
///
+ /// Returns `Err(ArrowError::InvalidArgumentError)` if `indices` is a
+ /// non-null `UInt32Array` with any out-of-bounds value.
+ ///
/// # Example
/// ```
- /// # use arrow_array::{record_batch, UInt64Array};
+ /// # use arrow_array::{record_batch, UInt32Array};
Review Comment:
yes exactly, Ill explain why in the other comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6171978547
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6177500894
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176520201)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.79 13.5±0.09ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.46 3.2±0.01ms ? ?/sec 1.00 938.1±8.32µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.41 2.6±0.01ms ? ?/sec 1.00 775.3±1.22µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.20 1905.0±7.47µs ? ?/sec 1.00 453.4±3.67µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.5±0.05ms ? ?/sec 1.00 15.3±0.09ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.69 4.4±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.31 2.9±0.00ms ? ?/sec 1.00 1264.9±2.49µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.07 852.7±2.53µs ? ?/sec 1.00 793.5±0.76µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.37 12.7±0.10ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.51 2.2±0.01ms ? ?/sec 1.00 893.0±8.83µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.09 1607.9±3.77µs ? ?/sec 1.00 767.6±1.14µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.43 1087.1±4.09µs ? ?/sec 1.00 447.4±0.77µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.34 19.6±0.04ms ? ?/sec 1.00 14.6±0.10ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.37 3.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.53 1942.7±3.50µs ? ?/sec 1.00 1267.9±5.36µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 845.3±1.51µs ? ?/sec 1.00 785.1±0.59µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.02 90.3±0.43ms ? ?/sec 1.00 89.0±0.54ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.03 5.3±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.00 3.7±0.02ms ? ?/sec 1.05 3.8±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.03 89.6±0.37ms ? ?/sec 1.00 87.2±0.73ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.6±0.02ms ? ?/sec 1.02 5.8±0.02ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.6±0.02ms ? ?/sec 1.06 3.8±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 144.9±1.09ms ? ?/sec 1.00 145.2±1.15ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.15 22.6±0.04ms ? ?/sec 1.00 19.6±0.11ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.05 6.4±0.01ms ? ?/sec 1.00 6.1±0.01ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 5.0±0.03ms ? ?/sec 1.00 5.0±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.3±0.13ms ? ?/sec 1.03 4.4±0.03ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.3±0.10ms ? ?/sec 1.01 32.8±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 8.0±0.04ms ? ?/sec 1.01 8.0±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 4.9±0.02ms ? ?/sec 1.02 5.0±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 3.8±0.02ms ? ?/sec 1.06 4.0±0.02ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.76 13.4±0.08ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.44 3.2±0.01ms ? ?/sec 1.00 933.9±3.96µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.31 2.5±0.01ms ? ?/sec 1.00 768.7±1.12µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.18 1880.2±7.56µs ? ?/sec 1.00 449.8±0.38µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.34 20.4±0.09ms ? ?/sec 1.00 15.2±0.08ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.68 4.4±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.20 2.8±0.01ms ? ?/sec 1.00 1267.8±5.08µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.07 846.5±1.32µs ? ?/sec 1.00 791.0±0.69µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.69 12.7±0.08ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.57 2.2±0.01ms ? ?/sec 1.00 855.5±5.56µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.73 1306.9±2.46µs ? ?/sec 1.00 754.9±0.65µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.28 569.1±0.55µs ? ?/sec 1.00 445.9±0.38µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.35 19.7±0.08ms ? ?/sec 1.00 14.5±0.09ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.35 3.4±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.32 1674.3±2.93µs ? ?/sec 1.00 1264.9±1.84µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.08 850.1±0.72µs ? ?/sec 1.00 785.7±1.13µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 1.00 11.2±0.29ms ? ?/sec 1.12 12.5±0.07ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.19 30.2±0.21ms ? ?/sec 1.00 4.9±0.04ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 2.43 4.6±0.02ms ? ?/sec 1.00 1891.6±37.90µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.28 1688.3±6.83µs ? ?/sec 1.00 1322.8±2.26µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.00 827.4±1.08µs ? ?/sec 1.01 832.1±1.69µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.40 47.3±0.22ms ? ?/sec 1.00 33.8±0.25ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.12 6.9±0.01ms ? ?/sec 1.00 6.2±0.03ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.06 2.7±0.00ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1518.3±1.70µs ? ?/sec 1.10 1674.6±2.14µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 23.5±0.09ms ? ?/sec 1.24 29.2±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.01ms ? ?/sec 1.23 6.3±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.00ms ? ?/sec 1.11 3.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.12 2.3±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.9±0.08ms ? ?/sec 1.19 39.1±0.13ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.8±0.06ms ? ?/sec 1.10 7.5±0.07ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 3.0±0.02ms ? ?/sec 1.11 3.3±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.02ms ? ?/sec 1.11 2.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.74 13.4±0.21ms ? ?/sec 1.00 4.9±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 2.80 2.9±0.01ms ? ?/sec 1.00 1042.9±4.51µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 3.40 3.1±0.01ms ? ?/sec 1.00 925.4±1.01µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.31 811.3±1.21µs ? ?/sec 1.00 621.0±1.05µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.31 20.1±0.12ms ? ?/sec 1.00 15.4±0.07ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.72 4.1±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.70 3.4±0.01ms ? ?/sec 1.00 1268.2±1.79µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.14 1046.1±1.46µs ? ?/sec 1.00 917.3±1.07µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.5±0.06ms ? ?/sec 1.34 18.1±0.09ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1558.0±8.35µs ? ?/sec 1.47 2.3±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 356.5±0.74µs ? ?/sec 1.75 624.4±1.38µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 240.6±0.29µs ? ?/sec 1.78 427.5±0.27µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.1±0.08ms ? ?/sec 1.28 25.7±0.12ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.32 3.3±0.01ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 616.8±1.33µs ? ?/sec 1.48 913.8±1.24µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 432.2±0.34µs ? ?/sec 1.45 626.3±0.65µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 4.74 12.8±0.29ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.78 1734.9±26.61µs ? ?/sec 1.00 974.5±3.75µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 701.9±4.12µs ? ?/sec 1.55 1090.2±3.04µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 448.3±1.43µs ? ?/sec 2.39 1070.5±11.75µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.52 19.3±0.28ms ? ?/sec 1.00 12.6±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.32 3.1±0.03ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1044.9±3.70µs ? ?/sec 1.39 1456.1±1.97µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 681.6±1.66µs ? ?/sec 1.66 1132.0±2.07µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 4.40 13.0±0.09ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 2.10 2.4±0.01ms ? ?/sec 1.00 1147.8±9.39µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.33 1933.2±2.65µs ? ?/sec 1.00 1457.7±1.64µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.28 1455.8±2.84µs ? ?/sec 1.00 1137.8±1.07µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.62 20.6±0.16ms ? ?/sec 1.00 12.8±0.09ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.46 3.7±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.22 2.3±0.00ms ? ?/sec 1.00 1856.3±2.32µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.18 1692.7±2.78µs ? ?/sec 1.00 1435.0±7.58µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.9±0.26ms ? ?/sec 1.15 40.0±0.25ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.0±0.03ms ? ?/sec 1.15 5.8±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1938.8±5.54µs ? ?/sec 1.15 2.2±0.00ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1218.8±2.35µs ? ?/sec 1.18 1436.6±12.11µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 58.2±0.21ms ? ?/sec 1.11 64.6±0.28ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.9±0.02ms ? ?/sec 1.06 11.5±0.03ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.01 5.3±0.01ms ? ?/sec 1.00 5.3±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.1±0.14ms ? ?/sec 1.14 50.2±0.18ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.5±0.09ms ? ?/sec 1.05 9.9±0.06ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.2±0.01ms ? ?/sec 1.06 5.5±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.7±0.01ms ? ?/sec 1.07 3.9±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 51.4±0.13ms ? ?/sec 1.10 56.8±0.15ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 10.6±0.05ms ? ?/sec 1.08 11.5±0.21ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 6.0±0.06ms ? ?/sec 1.05 6.3±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.3±0.03ms ? ?/sec 1.07 4.6±0.02ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 35.9±0.14ms ? ?/sec 1.24 44.6±0.17ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.1±0.02ms ? ?/sec 1.10 13.4±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.3±0.15ms ? ?/sec 1.00 11.4±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.3±0.18ms ? ?/sec 1.03 10.6±0.08ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.5±0.14ms ? ?/sec 1.16 54.1±0.12ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.5±0.04ms ? ?/sec 1.11 15.0±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.1±0.06ms ? ?/sec 1.04 10.5±0.05ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 8.6±0.05ms ? ?/sec 1.07 9.1±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.70 13.1±0.13ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 2.88 2.9±0.01ms ? ?/sec 1.00 1018.0±7.61µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 2.96 2.6±0.01ms ? ?/sec 1.00 879.6±2.34µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.31 810.6±1.17µs ? ?/sec 1.00 619.0±0.44µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.31 20.1±0.15ms ? ?/sec 1.00 15.4±0.08ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.71 4.1±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.14 2.8±0.01ms ? ?/sec 1.00 1300.7±2.02µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.16 1057.0±2.11µs ? ?/sec 1.00 913.6±1.23µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1r5gco5q-5f4c8912-bp72
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6176520201-3406-cgzdt 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1415.3s |
| Peak memory | 61.2 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 1390.6s |
| CPU sys | 22.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1445.3s |
| Peak memory | 61.4 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 1418.4s |
| CPU sys | 21.3s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6176520201:finish:32f1c1d3eab75cf513d42c589882b8a3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226709936
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226706836
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6326926966
alright make sense to me. I can review https://github.com/apache/arrow-rs/pull/11782 when its ready. This PR is ready for another review when your ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6326294324
> > The precedent I saw was InProgressArray::copy_rows. It is a safe trait method with no documented precondition, and its impls call get_unchecked on offset..offset + len behind a SAFETY comment that relies on the caller ([primitive.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/primitive.rs#L167-L169), [byte_view.rs](https://github.com/apache/arrow-rs/blob/d58a0fce35b67836ac9b2e20a7efe47a584e0a86/arrow-select/src/coalesce/byte_view.rs#L411-L412)).
>
> +1 on this. Im not sure why we need to mark the entire method as unsafe if we check for invariants before calling it. since the method is private, users cannot directly shoot themselves in the foot with this
My rationale is that we should uphold the rust soundness pledge in all code, not just public APIs: https://github.com/apache/arrow-rs/blob/main/arrow/README.md#safety
I made a draft PR to try and remove the precident
- https://github.com/apache/arrow-rs/pull/11782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216105
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(coalesce): add BatchCoalescer::push_batch_interleaved [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4077965989
##########
arrow-select/src/coalesce.rs:
##########
Review Comment:
similar point here. we need to be able to expose this to the `inProgressArrays` trait so it can avoid this copy. ofc the generic path can still fall back to this.
##########
arrow-select/src/coalesce.rs:
##########
@@ -298,6 +299,43 @@ impl BatchCoalescer {
self.push_batch(taken_batch)
}
+ /// Push rows gathered from multiple [`RecordBatch`]es into the Coalescer.
+ ///
+ /// This is semantically equivalent to calling [`interleave_record_batch`]
+ /// followed by [`Self::push_batch`], but avoids allocating an intermediate
+ /// [`RecordBatch`].
+ ///
+ /// Each element of `indices` is a `(batch_index, row_index)` pair that
+ /// identifies a row in `batches`.
+ ///
+ /// # Example
+ /// ```
+ /// # use arrow_array::record_batch;
+ /// # use arrow_select::coalesce::BatchCoalescer;
+ /// let batch1 = record_batch!(("a", Int32, [1, 2, 3])).unwrap();
+ /// let batch2 = record_batch!(("a", Int32, [4, 5, 6])).unwrap();
+ /// let indices = vec![(0, 2), (1, 0), (0, 0), (1, 2)]; // rows: 3, 4, 1, 6
+ /// let mut coalescer = BatchCoalescer::new(batch1.schema(), 1000);
+ /// coalescer
+ /// .push_batch_interleaved(&[&batch1, &batch2], &indices)
+ /// .unwrap();
+ /// coalescer.finish_buffered_batch().unwrap();
+ /// let out = coalescer.next_completed_batch().unwrap();
+ /// let expected = record_batch!(("a", Int32, [3, 4, 1, 6])).unwrap();
+ /// assert_eq!(out, expected);
+ /// ```
+ pub fn push_batch_interleaved(
+ &mut self,
+ batches: &[&RecordBatch],
+ indices: &[(usize, usize)],
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let interleaved = interleave_record_batch(batches, indices)?;
+ self.push_batch(interleaved)
Review Comment:
we should probably expose this `inProgressArrays` so they can take advantage of it.
##########
arrow/benches/coalesce_kernels.rs:
##########
@@ -611,9 +611,99 @@ fn add_all_take_benchmarks(c: &mut Criterion) {
}
}
-criterion_group!(benches, add_all_filter_benchmarks, add_all_take_benchmarks);
+criterion_group!(
+ benches,
+ add_all_filter_benchmarks,
+ add_all_take_benchmarks,
+ add_all_interleave_benchmarks
+);
criterion_main!(benches);
+fn add_all_interleave_benchmarks(c: &mut Criterion) {
+ let batch_size = 8192;
+ let num_source_batches = 4;
+
+ let primitive_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("int32_val", DataType::Int32, true),
+ Field::new("float_val", DataType::Float64, true),
+ Field::new(
+ "timestamp_val",
+ DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into())),
+ true,
+ ),
+ ]));
+
+ let single_utf8view_schema = SchemaRef::new(Schema::new(vec![Field::new(
+ "value",
+ DataType::Utf8View,
+ true,
+ )]));
+
+ let mixed_fsb_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("fsb16_val", DataType::FixedSizeBinary(16), true),
+ Field::new("fsb32_val", DataType::FixedSizeBinary(32), true),
+ ]));
+
+ // Binary, boolean, and dictionary all go through the generic path; group
+ // them in one schema so a single benchmark covers the whole family.
+ let mixed_generic_schema = SchemaRef::new(Schema::new(vec![
+ Field::new("binary_val", DataType::Binary, true),
+ Field::new("bool_val", DataType::Boolean, true),
+ Field::new(
+ "dict_val",
+ DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8)),
+ true,
+ ),
+ ]));
+
+ for null_density in [0.0, 0.1] {
+ for selectivity in [0.001, 0.01, 0.1, 0.8] {
+ for scenario in [
Review Comment:
I may remove this. we may just want to benchmark. from my understanding filtering isnt really a component for the two functions we are trying to optimize.
##########
arrow-select/src/coalesce.rs:
##########
@@ -298,6 +299,43 @@ impl BatchCoalescer {
self.push_batch(taken_batch)
}
+ /// Push rows gathered from multiple [`RecordBatch`]es into the Coalescer.
+ ///
+ /// This is semantically equivalent to calling [`interleave_record_batch`]
+ /// followed by [`Self::push_batch`], but avoids allocating an intermediate
+ /// [`RecordBatch`].
+ ///
+ /// Each element of `indices` is a `(batch_index, row_index)` pair that
+ /// identifies a row in `batches`.
+ ///
+ /// # Example
+ /// ```
+ /// # use arrow_array::record_batch;
+ /// # use arrow_select::coalesce::BatchCoalescer;
+ /// let batch1 = record_batch!(("a", Int32, [1, 2, 3])).unwrap();
+ /// let batch2 = record_batch!(("a", Int32, [4, 5, 6])).unwrap();
+ /// let indices = vec![(0, 2), (1, 0), (0, 0), (1, 2)]; // rows: 3, 4, 1, 6
+ /// let mut coalescer = BatchCoalescer::new(batch1.schema(), 1000);
+ /// coalescer
+ /// .push_batch_interleaved(&[&batch1, &batch2], &indices)
+ /// .unwrap();
+ /// coalescer.finish_buffered_batch().unwrap();
+ /// let out = coalescer.next_completed_batch().unwrap();
+ /// let expected = record_batch!(("a", Int32, [3, 4, 1, 6])).unwrap();
+ /// assert_eq!(out, expected);
+ /// ```
+ pub fn push_batch_interleaved(
+ &mut self,
+ batches: &[&RecordBatch],
+ indices: &[(usize, usize)],
+ ) -> Result<(), ArrowError> {
Review Comment:
it may be akward needing to add tuples like this when instead we should maybe expect a slice
```suggestion
pub fn push_batch_interleaved(
&mut self,
batches: &[&RecordBatch],
indices: &[(usize, (usize,usize))], // array , (start,end)
) -> Result<(), ArrowError> {
```
🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176520077
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176518416)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a18ec726fe638c5112a246fd10bc2d7bdfc0d3c5)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6176518416:start:21d765325337e56ab198b2ab402f62f7 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219046890
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6219044770)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a002f6344e0ec90742c2870d74c5aeed792cca9f) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a002f6344e0ec90742c2870d74c5aeed792cca9f)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6219044770:start:3a3acc7aedfa319eb794f6647970a7ec -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6270104620
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6270102929)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (1818602ad10103a544d89c1decba5e1424329179) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..1818602ad10103a544d89c1decba5e1424329179)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (primitive|.*utf8view)'
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6270102929:start:7982a5ec767ec906f0af052299d93026 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6270102929
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take: (primitive|.*utf8view)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175801379
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175800199)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (4ebe938906728c7dc823c3444f55c5a0ad4a32e9) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..4ebe938906728c7dc823c3444f55c5a0ad4a32e9)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175800199:start:5117073e8153475938777337b1d74ff8 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274353571
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6274351136)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (41bbe160f8e526fb621f0e2de3ebdead08cf45d2) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..41bbe160f8e526fb621f0e2de3ebdead08cf45d2)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6274351136:start:980ae678c93f10c984042a2a579117cf -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4321676862
##########
arrow-select/src/coalesce.rs:
##########
@@ -809,6 +930,26 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &[u32],
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let buf = Buffer::from_slice_ref(indices);
+ let scalar: ScalarBuffer<u32> = ScalarBuffer::new(buf, 0, indices.len());
+ let indices_arr = UInt32Array::new(scalar, None);
+ let taken = take(source.as_ref(), &indices_arr, None)?;
+ self.set_source(Some(taken));
+ let result = self.copy_rows(0, indices.len());
+ self.set_source(None);
Review Comment:
yeah this is a good point, Initially this method was called in `push_batch_with_indices` but I reverted it as there were some performance issues related to non primitives and view types.
see https://github.com/apache/arrow-rs/issues/11567 for a bit more context.
I think its worth removing the gate and creating follow up issues/tickets for possible regressions. I have https://github.com/apache/arrow-rs/pull/11732 which will address the boolean regression, we should do similar things for the other datatypes (separate PR's each).
Ill remove the gate and re-run the benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292220989
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6292219011)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (e31420bf3b6f3959e2a083bfdfa7dc68376580c3) to 1357f25 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1357f25f26ca67634e992bdb35220b4ee59c780f..e31420bf3b6f3959e2a083bfdfa7dc68376580c3)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6292219011:start:b8897895693da3c166e2087ac714f8d2 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4325738290
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
+ fn copy_rows_by_indices(
+ &mut self,
+ source: &ArrayRef,
+ indices: &UInt32Array,
+ ) -> Result<(), ArrowError> {
+ if indices.is_empty() {
+ return Ok(());
+ }
+ let taken = take(source.as_ref(), indices, None)?;
+ self.set_source(Some(taken));
+ let result = self.copy_rows(0, indices.len());
+ self.set_source(None);
+ result
Review Comment:
With the `has_non_specialized_filter_columns` check gone, this default now runs for sparse selections on `Utf8`, `Boolean`, `FixedSizeBinary`, `Dictionary`, `List`, and the other column types without a specialized `copy_rows_by_indices`. The tests that reach `push_batch_with_index` use only `Int32` and `Utf8View` columns, so nothing checks this path yet. Could we add a test with a mixed schema, for example `Int32`, `Utf8`, `Boolean`, and `FixedSizeBinary` with nulls, that splits across `target_batch_size` and compares the output against `take_record_batch`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6176104001
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175111319)
**Target:** coalesce_kernels
Comparing rich-T-kid/coalescer-interleave (a766c827931e2b8132df458b3f4ed7aafe918303) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a766c827931e2b8132df458b3f4ed7aafe918303)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.91 13.6±0.10ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.97 3.3±0.01ms ? ?/sec 1.00 832.5±4.01µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.75 2.7±0.02ms ? ?/sec 1.00 707.7±0.68µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.77 1899.7±6.70µs ? ?/sec 1.00 398.5±0.53µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.35 20.5±0.10ms ? ?/sec 1.00 15.2±0.04ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.78 4.4±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.43 2.9±0.03ms ? ?/sec 1.00 1209.7±1.23µs ? ?/sec
take: mixed_binaryview (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.15 849.9±0.73µs ? ?/sec 1.00 737.2±0.55µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.49 12.7±0.11ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 2.84 2.3±0.01ms ? ?/sec 1.00 798.8±1.73µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 2.29 1602.2±4.10µs ? ?/sec 1.00 700.7±0.89µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 2.75 1089.3±4.56µs ? ?/sec 1.00 395.5±0.62µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.38 19.7±0.12ms ? ?/sec 1.00 14.3±0.03ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.42 3.5±0.02ms ? ?/sec 1.00 2.5±0.00ms ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.60 1929.2±3.88µs ? ?/sec 1.00 1205.6±2.18µs ? ?/sec
take: mixed_binaryview (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.16 842.2±1.02µs ? ?/sec 1.00 729.0±0.58µs ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.001 1.08 100.2±1.72ms ? ?/sec 1.00 93.0±2.28ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.01 1.03 5.2±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.1 1.02 3.7±0.02ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
take: mixed_dict, 8192, nulls: 0, selectivity: 0.8 1.00 1999.1±10.94µs ? ?/sec 1.00 2.0±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001 1.05 96.6±1.46ms ? ?/sec 1.00 92.3±1.62ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01 1.00 5.7±0.01ms ? ?/sec 1.00 5.7±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1 1.04 3.6±0.02ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
take: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.0±0.01ms ? ?/sec 1.01 2.0±0.01ms ? ?/sec
take: mixed_utf8 extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 4.96 150.5±2.98ms ? ?/sec 1.00 30.4±5.12ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.001 1.17 22.6±0.09ms ? ?/sec 1.00 19.3±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.01 1.06 6.3±0.02ms ? ?/sec 1.00 5.9±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.1 1.01 5.1±0.07ms ? ?/sec 1.00 5.0±0.07ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 4.4±0.10ms ? ?/sec 1.00 4.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 32.5±0.14ms ? ?/sec 1.00 32.4±0.06ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.01 7.9±0.02ms ? ?/sec 1.00 7.8±0.02ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.0±0.09ms ? ?/sec 1.00 5.0±0.10ms ? ?/sec
take: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 3.9±0.06ms ? ?/sec 1.04 4.1±0.06ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001 3.88 13.4±0.08ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01 3.90 3.2±0.01ms ? ?/sec 1.00 823.4±2.53µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1 3.55 2.5±0.01ms ? ?/sec 1.00 712.5±0.91µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8 4.68 1870.0±9.31µs ? ?/sec 1.00 399.2±1.52µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001 1.37 20.7±0.11ms ? ?/sec 1.00 15.0±0.04ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01 1.76 4.4±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1 2.33 2.8±0.03ms ? ?/sec 1.00 1195.5±1.59µs ? ?/sec
take: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8 1.15 846.2±0.98µs ? ?/sec 1.00 734.1±1.01µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001 4.91 12.6±0.09ms ? ?/sec 1.00 2.6±0.00ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01 3.00 2.2±0.01ms ? ?/sec 1.00 745.9±4.08µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1 1.89 1315.4±2.54µs ? ?/sec 1.00 697.3±0.76µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8 1.43 570.1±0.85µs ? ?/sec 1.00 399.8±2.09µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001 1.39 19.7±0.11ms ? ?/sec 1.00 14.2±0.03ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01 1.44 3.5±0.02ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1 1.39 1669.3±2.81µs ? ?/sec 1.00 1202.8±1.42µs ? ?/sec
take: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8 1.14 844.3±1.21µs ? ?/sec 1.00 741.6±1.81µs ? ?/sec
take: primitive extra_large_repeat, input: 8192, output: 131072, nulls: 0, biggest: Some(1024) 15.06 11.1±0.06ms ? ?/sec 1.00 734.9±12.51µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.001 6.43 30.1±0.22ms ? ?/sec 1.00 4.7±0.02ms ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.01 3.14 4.6±0.02ms ? ?/sec 1.00 1448.8±70.79µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.1 1.38 1629.7±4.17µs ? ?/sec 1.00 1181.4±2.17µs ? ?/sec
take: primitive, 8192, nulls: 0, selectivity: 0.8 1.18 827.8±0.98µs ? ?/sec 1.00 701.3±1.06µs ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.001 1.43 47.5±0.15ms ? ?/sec 1.00 33.1±0.20ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.01 1.19 7.2±0.12ms ? ?/sec 1.00 6.0±0.01ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.1 1.09 2.6±0.00ms ? ?/sec 1.00 2.4±0.00ms ? ?/sec
take: primitive, 8192, nulls: 0.1, selectivity: 0.8 1.00 1512.3±2.30µs ? ?/sec 1.02 1548.5±2.21µs ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.001 1.00 24.1±0.52ms ? ?/sec 1.18 28.4±0.14ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.01 1.00 5.2±0.06ms ? ?/sec 1.18 6.1±0.01ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.1 1.00 2.8±0.01ms ? ?/sec 1.05 2.9±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0, selectivity: 0.8 1.00 2.0±0.00ms ? ?/sec 1.04 2.1±0.00ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.001 1.00 33.9±0.58ms ? ?/sec 1.15 38.9±0.10ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.01 1.00 6.9±0.07ms ? ?/sec 1.10 7.6±0.04ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.9±0.02ms ? ?/sec 1.08 3.2±0.02ms ? ?/sec
take: single_binary, 8192, nulls: 0.1, selectivity: 0.8 1.00 2.2±0.01ms ? ?/sec 1.06 2.3±0.02ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.001 2.79 13.5±0.17ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.01 3.36 3.0±0.01ms ? ?/sec 1.00 884.7±8.74µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.1 4.06 3.1±0.01ms ? ?/sec 1.00 760.7±1.14µs ? ?/sec
take: single_binaryview, 8192, nulls: 0, selectivity: 0.8 1.73 812.2±1.59µs ? ?/sec 1.00 469.8±0.54µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.001 1.32 20.1±0.31ms ? ?/sec 1.00 15.3±0.15ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.01 1.79 4.1±0.04ms ? ?/sec 1.00 2.3±0.01ms ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.1 2.80 3.4±0.01ms ? ?/sec 1.00 1206.8±1.60µs ? ?/sec
take: single_binaryview, 8192, nulls: 0.1, selectivity: 0.8 1.34 1042.4±1.91µs ? ?/sec 1.00 778.0±0.99µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.001 1.00 13.9±0.12ms ? ?/sec 1.32 18.3±0.36ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.01 1.00 1599.3±11.03µs ? ?/sec 1.35 2.2±0.03ms ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.1 1.00 365.3±0.89µs ? ?/sec 1.27 464.4±4.22µs ? ?/sec
take: single_boolean, 8192, nulls: 0, selectivity: 0.8 1.00 241.4±0.48µs ? ?/sec 1.22 294.1±0.46µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.7±0.15ms ? ?/sec 1.24 25.7±0.37ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.01 1.00 2.5±0.01ms ? ?/sec 1.24 3.1±0.03ms ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.1 1.00 621.1±1.11µs ? ?/sec 1.25 779.2±3.48µs ? ?/sec
take: single_boolean, 8192, nulls: 0.1, selectivity: 0.8 1.00 431.6±0.44µs ? ?/sec 1.14 491.8±0.51µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.001 1.00 12.5±0.09ms ? ?/sec 1.33 16.6±0.07ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.01 1.00 1690.1±6.14µs ? ?/sec 1.45 2.5±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.1 1.00 713.9±1.86µs ? ?/sec 1.18 842.6±1.20µs ? ?/sec
take: single_fsb16, 8192, nulls: 0, selectivity: 0.8 1.00 467.0±0.51µs ? ?/sec 1.17 545.4±0.93µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.001 1.00 19.4±0.17ms ? ?/sec 1.22 23.8±0.04ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.1±0.02ms ? ?/sec 1.13 3.5±0.01ms ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.1 1.00 1036.2±5.95µs ? ?/sec 1.15 1188.0±2.08µs ? ?/sec
take: single_fsb16, 8192, nulls: 0.1, selectivity: 0.8 1.00 686.3±1.36µs ? ?/sec 1.17 800.9±0.78µs ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.001 1.00 13.0±0.06ms ? ?/sec 1.33 17.3±0.12ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.01 1.00 2.4±0.01ms ? ?/sec 1.37 3.3±0.02ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.1 1.00 1925.0±2.48µs ? ?/sec 1.09 2.1±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0, selectivity: 0.8 1.00 1450.4±5.73µs ? ?/sec 1.10 1592.4±7.26µs ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.001 1.00 20.9±0.16ms ? ?/sec 1.19 25.0±0.04ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.01 1.00 3.7±0.02ms ? ?/sec 1.16 4.3±0.01ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.1 1.00 2.2±0.01ms ? ?/sec 1.08 2.4±0.00ms ? ?/sec
take: single_fsb32, 8192, nulls: 0.1, selectivity: 0.8 1.00 1698.2±7.39µs ? ?/sec 1.08 1830.4±6.54µs ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.001 1.00 34.9±0.19ms ? ?/sec 1.14 39.6±0.15ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.01 1.00 5.1±0.03ms ? ?/sec 1.11 5.6±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.1 1.00 1914.0±4.57µs ? ?/sec 1.08 2.1±0.00ms ? ?/sec
take: single_fsl4, 8192, nulls: 0, selectivity: 0.8 1.00 1187.7±2.09µs ? ?/sec 1.08 1286.0±5.75µs ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.001 1.00 59.3±0.23ms ? ?/sec 1.08 63.8±0.20ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.0±0.04ms ? ?/sec 1.05 11.5±0.02ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.3±0.01ms ? ?/sec 1.05 5.5±0.01ms ? ?/sec
take: single_fsl4, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.6±0.01ms ? ?/sec 1.04 4.7±0.01ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.001 1.00 44.7±0.38ms ? ?/sec 1.10 48.9±0.13ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.01 1.00 9.5±0.06ms ? ?/sec 1.02 9.8±0.03ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.1 1.00 5.1±0.02ms ? ?/sec 1.05 5.3±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0, selectivity: 0.8 1.00 3.6±0.02ms ? ?/sec 1.04 3.8±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.001 1.00 52.2±0.38ms ? ?/sec 1.08 56.4±0.10ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.01 1.00 11.1±0.12ms ? ?/sec 1.02 11.3±0.09ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.1 1.00 5.9±0.02ms ? ?/sec 1.02 6.1±0.02ms ? ?/sec
take: single_list, 8192, nulls: 0.1, selectivity: 0.8 1.00 4.4±0.03ms ? ?/sec 1.03 4.5±0.02ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.001 1.00 35.8±0.32ms ? ?/sec 1.22 43.6±0.17ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.01 1.00 12.1±0.04ms ? ?/sec 1.08 13.1±0.03ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.1 1.00 11.4±0.17ms ? ?/sec 1.00 11.4±0.16ms ? ?/sec
take: single_utf8, 8192, nulls: 0, selectivity: 0.8 1.00 10.6±0.28ms ? ?/sec 1.01 10.7±0.22ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.001 1.00 46.3±0.34ms ? ?/sec 1.15 53.3±0.14ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.01 1.00 13.7±0.05ms ? ?/sec 1.08 14.7±0.04ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.1 1.00 10.2±0.08ms ? ?/sec 1.02 10.4±0.06ms ? ?/sec
take: single_utf8, 8192, nulls: 0.1, selectivity: 0.8 1.00 8.8±0.30ms ? ?/sec 1.03 9.1±0.13ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.001 2.81 13.2±0.12ms ? ?/sec 1.00 4.7±0.04ms ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.01 3.12 2.9±0.01ms ? ?/sec 1.00 932.5±2.51µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.1 3.20 2.5±0.01ms ? ?/sec 1.00 793.1±2.03µs ? ?/sec
take: single_utf8view, 8192, nulls: 0, selectivity: 0.8 1.73 804.2±1.52µs ? ?/sec 1.00 464.5±0.62µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001 1.33 20.1±0.30ms ? ?/sec 1.00 15.1±0.05ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01 1.79 4.1±0.03ms ? ?/sec 1.00 2.3±0.00ms ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1 2.29 2.7±0.01ms ? ?/sec 1.00 1197.4±3.31µs ? ?/sec
take: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8 1.35 1049.9±1.37µs ? ?/sec 1.00 780.5±0.91µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>coalesce_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1bvfq8zp-52bcc797-2c94
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6175111319-3396-q6669 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1415.3s |
| Peak memory | 60.5 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 1387.6s |
| CPU sys | 21.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1415.3s |
| Peak memory | 60.0 MiB |
| Avg memory | 20.0 MiB |
| CPU user | 1376.8s |
| CPU sys | 34.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175111319:finish:7fd142f8de00659299e64a43b3a45b3d -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175113126
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175111319)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a766c827931e2b8132df458b3f4ed7aafe918303) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a766c827931e2b8132df458b3f4ed7aafe918303)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6175111319:start:7dd4ea6173739ab8557b497444f3ee10 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6175111319
run benchmark coalesce_kernels
env:
BENCH_FILTER: "take"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226711250
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6226709936)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (a01b4ffe081256a6f39cd26b5d1966864ddfe1b1) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..a01b4ffe081256a6f39cd26b5d1966864ddfe1b1)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6226709936:start:4a59d964491d808daf1c66a40233d2f5 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241217337
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216105)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (40874968ff2a73c735b384cd46f3bb24f191e46c) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..40874968ff2a73c735b384cd46f3bb24f191e46c)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6241216105:start:09567ba5e1f3a1f669d76f311f64a971 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): route push_batch_with_indices through zero-copy fast path [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241218144
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11174#issuecomment-6241216906)
**Target:** coalesce_kernels
**Sharding factor: 1** (1 workers).
Comparing rich-T-kid/coalescer-interleave (40874968ff2a73c735b384cd46f3bb24f191e46c) to 0706d31 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/0706d317a94098cec4b631031efcdf16f51ed1f5..40874968ff2a73c735b384cd46f3bb24f191e46c)
<details><summary>Run configuration</summary>
```yaml
run benchmark coalesce_kernels
env:
BENCH_FILTER: take
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6241216906:start:ac27594773c8ac65182e6d499cbdf21e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4329829747
##########
arrow-select/src/coalesce.rs:
##########
@@ -336,11 +404,81 @@ impl BatchCoalescer {
batch: RecordBatch,
indices: &dyn Array,
) -> Result<(), ArrowError> {
- // todo: optimize this to avoid materializing (copying the results of take indices to a new batch)
+ if let Some(u32_arr) = indices.as_any().downcast_ref::<UInt32Array>()
+ && u32_arr.null_count() == 0
+ {
+ validate_indices_bounds(u32_arr.values(), batch.num_rows())?;
Review Comment:
the only reason why this is kept to u32's it to avoid adding a macro that handles all types (`u8`...`u32`..`u64`). this can be added very easily but I think u32's are a fair bound
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(coalesce): directly copy sparse UInt32 selections [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11174:
URL: https://github.com/apache/arrow-rs/pull/11174#discussion_r4330285198
##########
arrow-select/src/coalesce.rs:
##########
@@ -853,6 +991,23 @@ trait InProgressArray: std::fmt::Debug + Send + Sync {
}
}
+ /// Copy rows at `indices` from `source` into the in-progress array.
+ /// `indices` must already be bounds-checked against `source`.
Review Comment:
ive updaated
```rust
let taken = take(source.as_ref(), indices, None)?;
```
to pass in options so that it will check the bounds before accessing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org