You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2026/02/18 08:33:24 UTC
[PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Fokko opened a new pull request, #9426:
URL: https://github.com/apache/arrow-rs/pull/9426
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Closes #9425.
# Rationale for this change
I noticed that this method is available on PrimitiveTypeBuilder, but missing on the GenericByteBuilder, which make sense since the gain is less, but after benchmarking, it shows a solid 10%:
```
┌───────────────────┬────────────────┬───────────────────┬─────────┐
│ Benchmark │ append_value_n │ append_value loop │ Speedup │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100/len=5 │ 371 ns │ 408 ns │ 10% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100/len=30 │ 456 ns │ 507 ns │ 10% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100/len=1024 │ 1.81 µs │ 1.95 µs │ 8% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=1000/len=5 │ 2.39 µs │ 2.87 µs │ 17% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=1000/len=30 │ 3.41 µs │ 3.89 µs │ 12% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=1000/len=1024 │ 12.3 µs │ 14.4 µs │ 15% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=10000/len=5 │ 23.8 µs │ 29.3 µs │ 19% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=10000/len=30 │ 33.7 µs │ 39.0 µs │ 14% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=10000/len=1024 │ 115.9 µs │ 135.0 µs │ 14% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100000/len=5 │ 227.5 µs │ 278.6 µs │ 18% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100000/len=30 │ 328.1 µs │ 377.9 µs │ 13% │
├───────────────────┼────────────────┼───────────────────┼─────────┤
│ n=100000/len=1024 │ 1.16 ms │ 1.34 ms │ 14% │
└───────────────────┴────────────────┴───────────────────┴─────────┘
```
I think this is still worthwhile to be added. Let me know what the community thinks!
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
A new public API.
# Are these changes tested?
Yes!
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
# Are there any user-facing changes?
A new public API.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2823833372
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
are there methods can we can use to reserve the capacity for value_builder and offsets_builder ahead of time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#issuecomment-3931456838
Out of curiosity, are there any benchmarks in the repo that show these improvements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832742355
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
My friend Claude created a benchmark:
```rust
cat arrow-array/benches/byte_builder.rs
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
use arrow_array::builder::StringBuilder;
use criterion::*;
use std::hint;
fn bench_append_value(c: &mut Criterion) {
let mut group = c.benchmark_group("append_value");
for &str_len in &[5, 30, 1024] {
let value = "x".repeat(str_len);
for &n in &[100, 1000, 10000, 100000] {
group.throughput(Throughput::Elements(n as u64));
group.bench_with_input(
BenchmarkId::new(format!("n={n}"), format!("len={str_len}")),
&(&value, n),
|b, &(value, n)| {
b.iter(|| {
let mut builder = StringBuilder::new();
for _ in 0..n {
builder.append_value(value);
}
hint::black_box(builder.finish());
})
},
);
}
}
group.finish();
}
fn bench_append_value_n(c: &mut Criterion) {
let mut group = c.benchmark_group("append_value_n");
for &str_len in &[5, 30, 1024] {
let value = "x".repeat(str_len);
for &n in &[100, 1000, 10000, 100000] {
group.throughput(Throughput::Elements(n as u64));
group.bench_with_input(
BenchmarkId::new(format!("n={n}"), format!("len={str_len}")),
&(&value, n),
|b, &(value, n)| {
b.iter(|| {
let mut builder = StringBuilder::new();
builder.append_value_n(value, n);
hint::black_box(builder.finish());
})
},
);
}
}
group.finish();
}
criterion_group!(benches, bench_append_value, bench_append_value_n);
criterion_main!(benches);
```
Let me know if you find this valuable to add to the repository.
This resulted in:
```
append_value_n improvements with reserve():
┌────────────────────┬─────────┬─────────┬─────────────┐
│ Config │ Before │ After │ Improvement │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=5 │ 2.96 µs │ 2.68 µs │ ~9% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=5 │ 29.3 µs │ 26.4 µs │ ~10% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=5 │ 268 µs │ 249 µs │ ~7% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100, len=30 │ 628 ns │ 548 ns │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=30 │ 4.22 µs │ 3.68 µs │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=30 │ 43.9 µs │ 36.3 µs │ ~17% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=30 │ 404 µs │ 350 µs │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100, len=1024 │ 3.16 µs │ 1.93 µs │ ~39% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=1024 │ 13.7 µs │ 11.1 µs │ ~19% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=1024 │ 729 µs │ 103 µs │ ~86% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=1024 │ 8.68 ms │ 1.02 ms │ ~88% │
└────────────────────┴─────────┴─────────┴─────────────┘
```
Keep in mind that the benchmark uses `::new` rather than `::with_capacity`. We can see that `.reserve` offers some benefits if you don't pre-allocate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2831259103
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
I didn't look closely at Rust, but from c++ there could be a potential footgun (copied the reference below). So calling append_n with n=1 in a loop was my concern if we ended up using reserve. It sounds like Rust might might make stronger guarantees (but there is wiggle room)
> Correctly using reserve() can prevent unnecessary reallocations, but inappropriate uses of reserve() (for instance, calling it before every [push_back()](https://en.cppreference.com/w/cpp/container/vector/push_back.html) call) may actually increase the number of reallocations (by causing the capacity to grow linearly rather than exponentially) and result in increased computational complexity and decreased performance. For example, a function that receives an arbitrary vector by reference and appends elements to it should usually not call reserve() on the vector, since it does not know of the vector's usage characteristics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2831217205
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
Is that directly related to this PR? Or a more general observation?
AFAIK, rust container allocations _are_ amortized, e.g. [Vec](https://doc.rust-lang.org/std/vec/struct.Vec.html#guarantees) docs state:
> Vec does not guarantee any particular growth strategy when reallocating when full, nor when reserve is called. The current strategy is basic and it may prove desirable to use a non-constant growth factor. Whatever strategy is used will of course guarantee O(1) amortized push.
So hopefully that's enough to avoid quadratic silliness even if we don't reserve, leaving only constant factors on the table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich merged PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#issuecomment-3932401786
> Out of curiosity, are there any benchmarks in the repo that show these improvements?
I've asked by friend Claude to generate a microbenchmark, where the result can be found in the PR description. I can commit the benchmark as well, but I figured that would generate a lot of _very_ specific benchmarks which won't be used that much, I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2823837417
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
(reserve is potentially dangerous performance wise if it doesn't do amortized allocations, or we would need to do the amortization here.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832052535
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
OK, so lets avoid adding it here. It seems better placed in caller code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2831268073
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
Or put another way, callers that care about it can probably call reserve themselves externally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832019486
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
You can reserve capacity by calling `with_capacity`:
https://github.com/apache/arrow-rs/blob/2f40f78e4feae3aee261d9608cede9535e1429e0/arrow-array/src/builder/generic_bytes_builder.rs#L43-L57
We already do this in our code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832229833
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
As this is `append_value_n` I believe it makes sense to add a `reserve here`.
While the builder provides `with_capacity` - the primary use case for using a builder is when you don't know the capacity upfront (in most other cases you can build the array from iterator or slice directly and skipping the builder overhead).
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
As this is `append_value_n` I believe it makes sense to add a `reserve`.
While the builder provides `with_capacity` - the primary use case for using a builder is when you don't know the capacity upfront (in most other cases you can build the array from iterator or slice directly and skipping the builder overhead).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832034855
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
+ self.offsets_builder.push(self.next_offset());
+ }
+ self.null_buffer_builder.append_n_non_nulls(n);
Review Comment:
Pulling this out of the loop gives the 10-20% speedup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832214642
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
+ self.offsets_builder.push(self.next_offset());
Review Comment:
If you want to make it faster:
`self.offsets_builder.append_trusted_len_iter(` could be used outside of the loop with an offset iterator.
IMO, also a `reserve` should be added here as this is a builder API - it makes not really sense in most cases to use a builder if you know the capacity upfront (you can build the array directly, which will be faster anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `append_value_n` to GenericByteBuilder [arrow-rs]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #9426:
URL: https://github.com/apache/arrow-rs/pull/9426#discussion_r2832742355
##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -110,6 +110,19 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
self.offsets_builder.push(self.next_offset());
}
+ /// Appends a value of type `T` into the builder `n` times.
+ ///
+ /// See [`Self::append_value`] for more panic information.
+ #[inline]
+ pub fn append_value_n(&mut self, value: impl AsRef<T::Native>, n: usize) {
+ let bytes = value.as_ref().as_ref();
+ for _ in 0..n {
+ self.value_builder.extend_from_slice(bytes);
Review Comment:
My friend Claude created a benchmark:
```rust
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
use arrow_array::builder::StringBuilder;
use criterion::*;
use std::hint;
fn bench_append_value(c: &mut Criterion) {
let mut group = c.benchmark_group("append_value");
for &str_len in &[5, 30, 1024] {
let value = "x".repeat(str_len);
for &n in &[100, 1000, 10000, 100000] {
group.throughput(Throughput::Elements(n as u64));
group.bench_with_input(
BenchmarkId::new(format!("n={n}"), format!("len={str_len}")),
&(&value, n),
|b, &(value, n)| {
b.iter(|| {
let mut builder = StringBuilder::new();
for _ in 0..n {
builder.append_value(value);
}
hint::black_box(builder.finish());
})
},
);
}
}
group.finish();
}
fn bench_append_value_n(c: &mut Criterion) {
let mut group = c.benchmark_group("append_value_n");
for &str_len in &[5, 30, 1024] {
let value = "x".repeat(str_len);
for &n in &[100, 1000, 10000, 100000] {
group.throughput(Throughput::Elements(n as u64));
group.bench_with_input(
BenchmarkId::new(format!("n={n}"), format!("len={str_len}")),
&(&value, n),
|b, &(value, n)| {
b.iter(|| {
let mut builder = StringBuilder::new();
builder.append_value_n(value, n);
hint::black_box(builder.finish());
})
},
);
}
}
group.finish();
}
criterion_group!(benches, bench_append_value, bench_append_value_n);
criterion_main!(benches);
```
Let me know if you find this valuable to add to the repository.
This resulted in:
```
append_value_n improvements with reserve():
┌────────────────────┬─────────┬─────────┬─────────────┐
│ Config │ Before │ After │ Improvement │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=5 │ 2.96 µs │ 2.68 µs │ ~9% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=5 │ 29.3 µs │ 26.4 µs │ ~10% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=5 │ 268 µs │ 249 µs │ ~7% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100, len=30 │ 628 ns │ 548 ns │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=30 │ 4.22 µs │ 3.68 µs │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=30 │ 43.9 µs │ 36.3 µs │ ~17% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=30 │ 404 µs │ 350 µs │ ~13% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100, len=1024 │ 3.16 µs │ 1.93 µs │ ~39% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=1000, len=1024 │ 13.7 µs │ 11.1 µs │ ~19% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=10000, len=1024 │ 729 µs │ 103 µs │ ~86% │
├────────────────────┼─────────┼─────────┼─────────────┤
│ n=100000, len=1024 │ 8.68 ms │ 1.02 ms │ ~88% │
└────────────────────┴─────────┴─────────┴─────────────┘
```
Keep in mind that the benchmark uses `::new` rather than `::with_capacity`. We can see that `.reserve` offers some benefits if you don't pre-allocate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org