You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/01/28 21:08:24 UTC
[PR] Improve `PrimitiveArray::from_iter` [arrow-rs]
Dandandan opened a new pull request, #9294:
URL: https://github.com/apache/arrow-rs/pull/9294
# Which issue does this PR close?
- Closes #NNN.
# Rationale for this change
Speeds up `from_iter`.
This speeds up creation for statistics if all values are present (common case):
```
Extract row group statistics for Int64/extract_statistics/Int64
time: [542.02 ns 543.41 ns 544.95 ns]
change: [−23.003% −22.574% −22.209%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
7 (7.00%) high mild
Extract data page statistics for Int64/extract_statistics/Int64
time: [8.9806 µs 8.9958 µs 9.0144 µs]
change: [−22.738% −21.961% −21.445%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) high mild
3 (3.00%) high severe
Extract row group statistics for UInt64/extract_statistics/UInt64
time: [523.80 ns 524.80 ns 526.02 ns]
change: [−25.064% −23.607% −22.464%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
```
# What changes are included in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#discussion_r2741412140
##########
arrow-array/src/array/primitive_array.rs:
##########
@@ -1467,13 +1468,14 @@ impl<T: ArrowPrimitiveType, Ptr: Into<NativeAdapter<T>>> FromIterator<Ptr> for P
.collect();
let len = null_builder.len();
+ let maybe_nulls = null_builder.finish().map(|x| x.into_inner().into_inner());
let data = unsafe {
ArrayData::new_unchecked(
T::DATA_TYPE,
len,
None,
- Some(null_builder.into()),
+ maybe_nulls,
Review Comment:
This makes sense, I think I saw heavy cloning in profiles even
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3819879447
Those are some pretty sweet results
```
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 70.8±0.41µs ? ?/sec 1.03 73.1±3.77µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.00 11.6±0.15µs ? ?/sec 1.33 15.4±0.30µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.00 13.2±0.17µs ? ?/sec 1.35 17.7±0.16µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.00 70.5±0.40µs ? ?/sec 1.02 72.2±0.60µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 11.8±0.41µs ? ?/sec 1.34 15.8±1.05µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 1054.8±8.28ns ? ?/sec 1.14 1205.3±20.01ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.00 538.6±7.15ns ? ?/sec 1.84 989.6±10.53ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.00 542.8±5.83ns ? ?/sec 1.81 979.8±12.40ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.00 1050.5±8.53ns ? ?/sec 1.11 1163.6±17.81ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.00 540.4±6.24ns ? ?/sec 1.82 985.5±11.18ns ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817947338
> run benchmark arrow_statistics
Sorry I think the VM runner got rebooted / wasn't working. I restarted it and now the queue is good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#discussion_r2741364810
##########
arrow-array/src/array/primitive_array.rs:
##########
@@ -1467,13 +1468,14 @@ impl<T: ArrowPrimitiveType, Ptr: Into<NativeAdapter<T>>> FromIterator<Ptr> for P
.collect();
let len = null_builder.len();
+ let maybe_nulls = null_builder.finish().map(|x| x.into_inner().into_inner());
let data = unsafe {
ArrayData::new_unchecked(
T::DATA_TYPE,
len,
None,
- Some(null_builder.into()),
+ maybe_nulls,
Review Comment:
I think you can make this even faster (avoid another allocation) by directly constructing the PrimitiveArray
```suggestion
let maybe_nulls = null_builder.finish();
PrimitiveArray::new(ScalarBuffer::from(buffer), maybe_nulls)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3819104947
🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing from_iter_speed (2a4ab06078460f88fe0d6b8b8c919f17251849a5) to bd76edd0f947330bde5b458b32f513f74bdbb1a6 [diff](https://github.com/apache/arrow-rs/compare/bd76edd0f947330bde5b458b32f513f74bdbb1a6..2a4ab06078460f88fe0d6b8b8c919f17251849a5)
BENCH_NAME=arrow_statistics
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_statistics
BENCH_FILTER=
BENCH_BRANCH_NAME=from_iter_speed
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3818328113
🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing from_iter_speed (2a4ab06078460f88fe0d6b8b8c919f17251849a5) to bd76edd0f947330bde5b458b32f513f74bdbb1a6 [diff](https://github.com/apache/arrow-rs/compare/bd76edd0f947330bde5b458b32f513f74bdbb1a6..2a4ab06078460f88fe0d6b8b8c919f17251849a5)
BENCH_NAME=arrow_statistics
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_statistics
BENCH_FILTER=
BENCH_BRANCH_NAME=from_iter_speed
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#discussion_r2741556032
##########
arrow-array/src/array/primitive_array.rs:
##########
@@ -1467,13 +1468,14 @@ impl<T: ArrowPrimitiveType, Ptr: Into<NativeAdapter<T>>> FromIterator<Ptr> for P
.collect();
let len = null_builder.len();
+ let maybe_nulls = null_builder.finish().map(|x| x.into_inner().into_inner());
let data = unsafe {
ArrayData::new_unchecked(
T::DATA_TYPE,
len,
None,
- Some(null_builder.into()),
+ maybe_nulls,
Review Comment:
Yeah it shaves off another ~30%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817591918
run benchmark arrow_statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3819145433
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group from_iter_speed main
----- --------------- ----
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 70.8±0.41µs ? ?/sec 1.03 73.1±3.77µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.00 11.6±0.15µs ? ?/sec 1.33 15.4±0.30µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.00 13.2±0.17µs ? ?/sec 1.35 17.7±0.16µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.00 70.5±0.40µs ? ?/sec 1.02 72.2±0.60µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 11.8±0.41µs ? ?/sec 1.34 15.8±1.05µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 1054.8±8.28ns ? ?/sec 1.14 1205.3±20.01ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.00 538.6±7.15ns ? ?/sec 1.84 989.6±10.53ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.00 542.8±5.83ns ? ?/sec 1.81 979.8±12.40ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.00 1050.5±8.53ns ? ?/sec 1.11 1163.6±17.81ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.00 540.4±6.24ns ? ?/sec 1.82 985.5±11.18ns ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817948058
🤖 Hi @alamb, you asked to view the benchmark queue (https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817947944).
| Job | User | Benchmarks | Comment |
| --- | --- | --- | --- |
| `20055_3815459655.sh` | Dandandan | default | `https://github.com/apache/datafusion/pull/20055#issuecomment-3815459655` |
| `20055_3815478475.sh` | Dandandan | default | `https://github.com/apache/datafusion/pull/20055#issuecomment-3815478475` |
| `arrow-9294-3817591918.sh` | Dandandan | arrow_statistics | `https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817591918` |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3818363174
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group from_iter_speed main
----- --------------- ----
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.12 82.1±0.98µs ? ?/sec 1.00 73.5±0.73µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.00 12.2±0.18µs ? ?/sec 1.27 15.6±0.08µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.00 14.0±0.24µs ? ?/sec 1.26 17.6±0.52µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.12 81.5±0.48µs ? ?/sec 1.00 72.8±0.31µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 12.3±0.04µs ? ?/sec 1.25 15.5±0.13µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 1015.3±12.19ns ? ?/sec 1.15 1167.6±15.76ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.00 535.3±4.27ns ? ?/sec 1.81 967.6±10.35ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.00 537.1±7.38ns ? ?/sec 1.80 966.8±24.59ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.00 1006.9±32.83ns ? ?/sec 1.12 1123.8±16.19ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.00 535.2±10.28ns ? ?/sec 1.81 969.5±8.96ns ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3818857678
run benchmark arrow_statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#issuecomment-3817947944
show benchmark queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve `PrimitiveArray::from_iter` perf [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9294:
URL: https://github.com/apache/arrow-rs/pull/9294#discussion_r2741467279
##########
arrow-array/src/array/primitive_array.rs:
##########
@@ -1467,13 +1468,14 @@ impl<T: ArrowPrimitiveType, Ptr: Into<NativeAdapter<T>>> FromIterator<Ptr> for P
.collect();
let len = null_builder.len();
+ let maybe_nulls = null_builder.finish().map(|x| x.into_inner().into_inner());
let data = unsafe {
ArrayData::new_unchecked(
T::DATA_TYPE,
len,
None,
- Some(null_builder.into()),
+ maybe_nulls,
Review Comment:
BTW I made another small PR
- https://github.com/apache/arrow-rs/pull/9299
As well as filed an epic to track trying to avoid ArrayData when possible:
- https://github.com/apache/arrow-rs/issues/9298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org