You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/06/30 12:27:21 UTC
[I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
alamb opened a new issue, #10245:
URL: https://github.com/apache/arrow-rs/issues/10245
### Is your feature request related to a problem or challenge?
As @Jefffrey has observed in https://github.com/apache/arrow-rs/pull/10229#discussion_r3496012625, switching from `OffsetBufferBuilder` / `BufferBuilder` to use `Vec<..>` often results in a non trivial speedup.
This is likely because the Rust language has a *highly* optimized Vec implementation
### Describe the solution you'd like
Basically try and change any code like this
```rust
let mut values = BufferBuilder::<T::Native>::new(len);
```
To use
```rust
let mut values = Vec<T::Native>::with_capacity(len);
```
And various equivalent
## Note to contributors:
please make targeted PRs (that change one or two usages each) to keep the PRs easy to review and thus fast to merge
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5586565493
I updated the checklist in the description to reflect current `main`:
- `arrow-cast` (#10867), `arrow-ipc` (#11005), and `arrow-row` (#10851) are done ✅
- `arrow-avro` is marked as won't-do per the discussion in #10923 (keeping `OffsetBufferBuilder` there is clearer and it isn't a tight hot loop)
- Two callsites remain, both with open PRs:
- `arrow-arith/src/arity.rs` `try_binary` nullable path (#10518)
- `arrow-select/src/zip.rs` byte-array path (#10913)
Once those two PRs are merged (or otherwise resolved) we can close this epic 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "cakeni (via GitHub)" <gi...@apache.org>.
cakeni commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5252478542
I'd like to take the two BufferBuilder usages in arrow-select/src/take.rs's run-end encoded take path and replace them with Vec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "zhangxinyao88 (via GitHub)" <gi...@apache.org>.
zhangxinyao88 commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5428885163
I’d like to take the `arrow-cast/src/cast/mod.rs::cast_byte_container` callsite and replace its `BufferBuilder<TO::Offset>` usage with `Vec<TO::Offset>` in a focused PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "zhangxinyao88 (via GitHub)" <gi...@apache.org>.
zhangxinyao88 commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5473882601
I’d like to take the remaining `arrow-avro/src/reader/record.rs` Decoder callsites. I have a working prototype replacing the long-lived `OffsetBufferBuilder<i32>` state with `Vec<i32>`; the arrow-avro tests and Clippy pass. Initial benchmarks show improvements for Binary and Map, with String and Array remaining neutral. I’ll prepare this as a focused one-file PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "cakeni (via GitHub)" <gi...@apache.org>.
cakeni commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5252131462
I'd like to take the remaining BufferBuilder usage in arrow-json's MapArrayDecoder and replace it with Vec, following the existing ListArrayDecoder approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-4981058013
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "EltonChang1 (via GitHub)" <gi...@apache.org>.
EltonChang1 commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5457770500
I’d like to take the `arrow-ipc/src/writer.rs::into_zero_offset_run_array` callsite and replace its `BufferBuilder<R::Native>` usage with `Vec<R::Native>` in a focused PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "zhangxinyao88 (via GitHub)" <gi...@apache.org>.
zhangxinyao88 commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5418842448
I’d like to take the `arrow-row/src/variable.rs::decode_binary` callsite and replace its `BufferBuilder<I>` usage with `Vec<I>` in a focused PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "cakeni (via GitHub)" <gi...@apache.org>.
cakeni commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5252478206
I'd like to take the remaining BufferBuilder usage in arrow-ord/src/sort.rs and replace it with Vec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-4845428823
Thanks @alamb adding https://github.com/apache/datafusion/pull/23195 as the reference where `Vec` replaces `OffsetBufferBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5429533501
tagging this since its similar - https://github.com/apache/arrow-rs/issues/10868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "alamb (via GitHub) via github" <gi...@arrow.apache.org>.
alamb closed issue #10245: [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible
URL: https://github.com/apache/arrow-rs/issues/10245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5174034547
@mzabaluev @Anakin100100 are either of you currently working on this? Im working on a few other issues so im happy to untake, otherwise Ill push up a PR for this tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5253433187
nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "yashkuceriya (via GitHub)" <gi...@apache.org>.
yashkuceriya commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5465229296
I would like to take the remaining `arrow-select/src/zip.rs` byte-array callsite and replace its `OffsetBufferBuilder` usage with `Vec` in a focused PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "ax1s-x1zz (via GitHub)" <gi...@apache.org>.
ax1s-x1zz commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5559312112
Opened PR #11005 to cover the [](https://github.com/apache/arrow-rs/blob/main/arrow-ipc/src/writer.rs#L1280) callsite (run-ends re-encoding). Unlike the other remaining callsites in your list, that one still used on current main; and had already been converted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "WaterWhisperer (via GitHub) via github" <gi...@arrow.apache.org>.
WaterWhisperer commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-6706270952
> I updated the checklist in the description to reflect current `main`:
>
> * Two callsites remain, both with open PRs:
>
> * `arrow-arith/src/arity.rs` `try_binary` nullable path ([perf: optimize loops for primitive types binary kernels. #10518](https://github.com/apache/arrow-rs/pull/10518))
> * `arrow-select/src/zip.rs` byte-array path ([Replace OffsetBufferBuilder with Vec in scalar byte-array zip #10913](https://github.com/apache/arrow-rs/pull/10913))
>
> Once those two PRs are merged (or otherwise resolved) we can close this epic 🎉
I looked into the remaining nullable `try_binary` callsite referenced in #10518.
The local patch only replaces `BufferBuilder` with `vec![O::Native::default(); len]` and converts the result with `.into()`.
A focused benchmark on base commit showed repeatable regressions in some cases. It calls `try_binary` directly with checked addition, fixed seeded inputs, and nulls on the left input only. Input generation is outside the timed operation.
Environment: Intel i7-13700H, Linux x86_64, Rust 1.99.1, glibc 2.39, default release settings and system allocator. Both versions were pinned to the same CPU core, with the powersave governor unchanged. Each case was measured six times per version, alternating execution order and shuffling case order between rounds, using 100 Criterion samples and a 2-second measurement target per run.
Selected results below are average execution times across the six runs:
- Decimal256, 8,192 rows, 10% nulls: 30.03 → 33.33 µs, **11.0% slower**.
- Decimal256, 65,536 rows, 10% nulls: 284.00 → 305.35 µs, **7.5% slower**.
- Decimal256, 1,048,576 rows, 10% nulls: 17.42 → 17.68 ms, **1.4% slower**.
- Int32, 128 rows, all null: 87.64 → 101.36 ns, **15.7% slower**.
For Decimal256 with 50% nulls, the corresponding changes at those three batch sizes were +10.3%, +6.5%, and +0.8%. Larger Int32 cases were generally close to unchanged.
The Decimal256 regressions at 8,192 and 65,536 rows occurred in all six comparisons. At one million rows, the differences were much smaller. One unchanged no-null Decimal256 control also showed an approximately 10% swing in one comparison, so I would treat small differences cautiously.
These results are limited to this machine, toolchain, and workload. The nullable benchmark mentioned in #10518 was not included in that PR, so I cannot compare the two setups directly or explain its reported gains conclusively.
Would it make sense to keep `BufferBuilder` at this callsite for now, pending results from other environments or a replacement with more consistent benefits? I have retained the focused benchmark and raw results locally and can share them for reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "cakeni (via GitHub)" <gi...@apache.org>.
cakeni commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5252478857
I'd like to take the remaining BufferBuilder usages in arrow-string and replace them with Vec, split into focused PRs for concat_elements.rs and substring.rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "mzabaluev (via GitHub)" <gi...@apache.org>.
mzabaluev commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5179285590
@Rich-T-kid I don't have anything pending on this, working on other things now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "kowanietz (via GitHub)" <gi...@apache.org>.
kowanietz commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5369084632
I'll take the remaining BufferBuilder usage in `arrow-select/src/take.rs` fixed-size binary take path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-5416212235
I had claude code find other uses that we could migrate. The list is below (and I added it to the ticket description)
- [ ] [`arrow-arith/src/arity.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-arith/src/arity.rs): [`try_binary` nullable path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-arith/src/arity.rs#L285) builds the output values with `BufferBuilder::<O::Native>` (likely covered by #10518)
- [ ] [`arrow-cast/src/cast/mod.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-cast/src/cast/mod.rs): [`cast_byte_container`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-cast/src/cast/mod.rs#L2835) converts offsets between offset types with `BufferBuilder::<TO::Offset>`
- [ ] [`arrow-ipc/src/writer.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-ipc/src/writer.rs): [`into_zero_offset_run_array`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-ipc/src/writer.rs#L1295) re-encodes sliced run-ends with `BufferBuilder::<R::Native>`
- [ ] [`arrow-row/src/variable.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/variable.rs): [`decode_binary`](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-row/src/variable.rs#L287) builds offsets with `BufferBuilder::<I>`
- [ ] [`arrow-select/src/zip.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-select/src/zip.rs): [byte-array zip path](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-select/src/zip.rs#L562) uses `OffsetBufferBuilder::<T::Offset>`
- [ ] [`arrow-avro/src/reader/record.rs`](https://github.com/apache/arrow-rs/blob/main/arrow-avro/src/reader/record.rs): [`Decoder` enum variants](https://github.com/apache/arrow-rs/blob/0e581556e8ba4ec4fff4d1e0ed11d44730b92767/arrow-avro/src/reader/record.rs#L245) (`Binary`/`String`/`StringView`/`Array`/`Map`) hold `OffsetBufferBuilder<i32>` as long-lived decoder state (~15 sites; a larger refactor than the others since the builders are not per-call scratch buffers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Epic] Replace `OffsetBufferBuilder` / `BufferBuilder` usage with `Vec`, when possible [arrow-rs]
Posted by "alamb (via GitHub) via github" <gi...@arrow.apache.org>.
alamb commented on issue #10245:
URL: https://github.com/apache/arrow-rs/issues/10245#issuecomment-6706902598
> Would it make sense to keep BufferBuilder at this callsite for now, pending results from other environments or a replacement with more consistent benefits? I have retained the focused benchmark and raw results locally and can share them for reproduction.
Yes I agree -- thank you @WaterWhisperer
Let's call this done for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org