You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/07/10 20:02:30 UTC
[PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Rich-T-kid opened a new pull request, #10317:
URL: https://github.com/apache/arrow-rs/pull/10317
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- works towards #9843.
- a follow PR will be made to wire up the fallible API into other portions of the code base.
# Rationale for this change
see #9843.
`MutableBuffer` is a low-level building block used throughout the Arrow implementation. Every growth operation, `reserve`, `resize`, `push`, `extend_from_slice`, etc, can panic on arithmetic overflow or an invalid allocation layout. In library code that needs to surface errors gracefully. This PR adds methods to return errors instead of panicking.
I made new methods instead of adding conditional checks/wrappers for each method because its easier to slowly roll these changes out to call sites instead of swapping.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
```
┌──────────────────────────┬──────────────────────┐
│ Fallible │ Infallible │
├──────────────────────────┼──────────────────────┤
│ try_reserve │ reserve │
├──────────────────────────┼──────────────────────┤
│ try_repeat_slice_n_times │ repeat_slice_n_times │
├──────────────────────────┼──────────────────────┤
│ try_reallocate (private) │ reallocate │
├──────────────────────────┼──────────────────────┤
│ try_resize │ resize │
├──────────────────────────┼──────────────────────┤
│ try_shrink_to_fit │ shrink_to_fit │
├──────────────────────────┼──────────────────────┤
│ try_extend_from_slice │ extend_from_slice │
├──────────────────────────┼──────────────────────┤
│ try_push │ push │
├──────────────────────────┼──────────────────────┤
│ try_extend_zeros │ extend_zeros │
└──────────────────────────┴──────────────────────┘
```
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are these changes tested?
Currently these methods aren't wired up to anything in the code base so I didn't include any test. Id be happy to include some. The implementations of the fallible API's are 1-1 to their non-fallible counter parts so i'm not sure if this would be needed.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
# Are there any user-facing changes?
yes, users can now work with errors instead of having their programs panic.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993530949
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4993496337-1112-76mf4 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=builder
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench builder
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031503539
Benchmark for [this request](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031452174) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
record_batch
regexp_kernels
row_format
row_group_index_reader
row_selection_cursor
row_selector
serde
sort_kernel
string_dictionary_builder
string_run_builder
string_run_iterator
substring_kernels
take_kernels
union_array
variant_builder
variant_kernels
variant_validation
view_types
writer_overhead
zip_kernels
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5058819085
> > > It uses an opaque struct rather than an enum, which means we are more free to adjust implementation without changing internals, making it a breaking api change.
> > it just came to mind that we can mark this enum as #[non_exhaustive]
> the error set is probably bounded enough to not require this i think? i.e. how many more error types would be possible in this context, to warrant this for the future
this is also true, no other error cases come to mind right now. ill revert the `#[non_exhaustive]` unless anyone feels strongly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056093032
> > It uses an opaque struct rather than an enum, which means we are more free to adjust implementation without changing internals, making it a breaking api change.
>
> it just came to mind that we can mark this enum as `#[non_exhaustive]`
the error set is probably bounded enough to not require this i think? i.e. how many more error types would be possible in this context, to warrant this for the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993610143
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=1024 1.00 3.2±0.03µs ? ?/sec 1.00 3.2±0.04µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=3 1.23 99.0±1.04ns ? ?/sec 1.00 80.6±0.77ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=64 1.03 314.5±2.48ns ? ?/sec 1.00 306.1±2.66ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=8192 1.00 24.8±0.18µs ? ?/sec 1.03 25.5±0.23µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=1024 1.00 2.1±0.03µs ? ?/sec 1.11 2.4±0.00µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=3 1.06 74.7±1.21ns ? ?/sec 1.00 70.2±0.89ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=64 1.00 240.3±1.75ns ? ?/sec 1.09 260.7±1.15ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=8192 1.00 16.3±0.60µs ? ?/sec 1.13 18.4±0.05µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=1024 1.00 2.2±0.08µs ? ?/sec 1.09 2.4±0.00µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=3 1.03 72.9±1.10ns ? ?/sec 1.00 70.7±0.28ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=64 1.00 213.9±2.66ns ? ?/sec 1.06 225.8±0.32ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=8192 1.00 17.5±0.83µs ? ?/sec 1.03 18.1±0.02µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=1024 1.00 1258.0±0.82ns ? ?/sec 1.00 1255.9±1.66ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=3 1.33 110.3±1.97ns ? ?/sec 1.00 82.9±0.41ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=64 1.06 209.6±0.69ns ? ?/sec 1.00 198.2±0.76ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=8192 1.00 10.5±0.01µs ? ?/sec 1.00 10.5±0.01µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=1024 1.03 363.9±0.55ns ? ?/sec 1.00 353.1±0.87ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=3 1.04 78.5±1.81ns ? ?/sec 1.00 75.6±0.61ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=64 1.01 153.9±0.62ns ? ?/sec 1.00 152.6±0.78ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=8192 1.00 2.0±0.00µs ? ?/sec 1.00 2.0±0.00µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=1024 1.17 213.7±0.68ns ? ?/sec 1.00 182.4±1.01ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=3 1.00 80.5±1.25ns ? ?/sec 1.02 82.1±1.11ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=64 1.09 113.4±2.68ns ? ?/sec 1.00 103.6±0.34ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=8192 1.08 436.8±1.55ns ? ?/sec 1.00 405.3±1.16ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 230.1s |
| Peak memory | 9.4 MiB |
| Avg memory | 5.7 MiB |
| CPU user | 227.9s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 240.1s |
| Peak memory | 8.5 MiB |
| Avg memory | 4.2 MiB |
| CPU user | 234.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031397972
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4970510257
> It doesn't duplicate the logic, i.e, the fallible versions are just try_<method>().unwrap() which means we don't need to adjust it in two places
This makes sense, this is a much cleaner pattern and it makes migration later down the line.
>It uses an opaque struct rather than an enum, which means we are more free to adjust implementation without changing internals, making it a breaking api change.
I can see how this may be a worry but I think the number of ways these functions can fail is very small. I think an enum is more applicable in this case. happy to hear other thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337
run benchmarks mutable_buffer_repeat_slice buffer_create builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056101839
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5056082827-1247-xldtx 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (5c48380bad1697ee95736956117393c715eca39e) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..5c48380bad1697ee95736956117393c715eca39e)
BENCH_NAME=builder
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench builder
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993573490
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
Buffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
MutableBuffer iter bitset 1.00 44.8±0.36ms ? ?/sec 1.00 45.0±0.32ms ? ?/sec
MutableBuffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.36 4.2±0.04ms ? ?/sec
from_slice 1.00 222.9±0.85µs ? ?/sec 1.14 254.7±1.21µs ? ?/sec
from_slice prepared 1.01 213.3±0.84µs ? ?/sec 1.00 211.7±0.77µs ? ?/sec
mutable 1.00 276.6±1.17µs ? ?/sec 1.00 277.7±4.23µs ? ?/sec
mutable extend 1.00 547.0±8.93µs ? ?/sec 1.00 546.9±0.78µs ? ?/sec
mutable iter extend_from_slice 1.01 1161.0±26.02µs ? ?/sec 1.00 1153.5±1.23µs ? ?/sec
mutable prepared 1.00 182.1±0.50µs ? ?/sec 1.01 184.2±0.79µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 118.8 MiB |
| Avg memory | 22.9 MiB |
| CPU user | 80.3s |
| CPU sys | 12.4s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 95.0s |
| Peak memory | 136.4 MiB |
| Avg memory | 23.5 MiB |
| CPU user | 77.5s |
| CPU sys | 13.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993529803
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4993496337-1111-9mrcl 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=buffer_create
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench buffer_create
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056111719
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5056082827-1245-zsp2p 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (5c48380bad1697ee95736956117393c715eca39e) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..5c48380bad1697ee95736956117393c715eca39e)
BENCH_NAME=mutable_buffer_repeat_slice
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench mutable_buffer_repeat_slice
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056111878
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5056082827-1246-mfs6k 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (5c48380bad1697ee95736956117393c715eca39e) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..5c48380bad1697ee95736956117393c715eca39e)
BENCH_NAME=buffer_create
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench buffer_create
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4970915729
@cetra3 could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3584180629
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
make sense, [2d0f74fe](https://github.com/apache/arrow-rs/pull/10317/commits/2d0f74e105ad91063819df2116e44f393ca9656b) takes a similar approach.
>I use a memory limited global allocator in some scenarios already
Im curious as to why? is this mostly for testing in low memory enviroments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5023448820
We have found boolean benchmarks to be quite sensitive -- allocation, alignment, etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5052897814
Benchmark for [this request](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5052882061) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
record_batch
regexp_kernels
row_format
row_group_index_reader
row_selection_cursor
row_selector
serde
sort_kernel
string_dictionary_builder
string_run_builder
string_run_iterator
substring_kernels
take_kernels
union_array
variant_builder
variant_kernels
variant_validation
view_types
writer_overhead
zip_kernels
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827
run benchmarks mutable_buffer_repeat_slice buffer_create builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031503259
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031452174)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5031452174-1197-wlf9k 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench sql_planner
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031416894
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031397972)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5031397972-1196-6nqz4 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench sql_planner
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031417161
Benchmark for [this request](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031397972) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
record_batch
regexp_kernels
row_format
row_group_index_reader
row_selection_cursor
row_selector
serde
sort_kernel
string_dictionary_builder
string_run_builder
string_run_iterator
substring_kernels
take_kernels
union_array
variant_builder
variant_kernels
variant_validation
view_types
writer_overhead
zip_kernels
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5081552296
do we need to merge main into this to fix the CI issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3587478608
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
nice, I also have interest in https://github.com/apache/datafusion/issues/22758 & https://github.com/apache/arrow-rs/issues/8938
> If we have a global allocator that internally limits how many allocations it dishes out, we can course correct in code if an error is returned
make sense.
Would the next step be to replace all OOM panics with error propagation? Off the top of my head, I think arrow-buffer's crate deals with memory management most directly out of all the crates, so it would be a good place to start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5013914602
> MutableBuffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.36 4.2±0.04ms ? ?/sec
> from_slice 1.00 222.9±0.85µs ? ?/sec 1.14 254.7±1.21µs ? ?/sec
> bench_bool/bench_bool 1.00 296.1±0.40µs 1688.8 MB/sec 1.22 361.2±0.21µs 1384.3 MB/sec
its strange that the only regression seems to come from the boolean benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993528614
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4993496337-1110-h485j 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=mutable_buffer_repeat_slice
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench mutable_buffer_repeat_slice
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993513688
> from the docs for [handle_alloc_error](https://doc.rust-lang.org/beta/std/alloc/fn.handle_alloc_error.html)
>
> > it may either panic (resulting in unwinding or aborting as per configuration for all panics), or abort the process (with no unwinding).
>
> it seems that once this branch is hit there isn't much for the process to do anyway? in both cases it aborts the process.
yeah I was mainly looking at this part of its doc:
> Callers of memory allocation APIs wishing to cease execution in response to an allocation error are encouraged to call this function, rather than directly invoking [panic!](https://doc.rust-lang.org/beta/core/macro.panic.html) or similar.
but its probably fine i guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3584134804
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
I.e, in my PR I had a simple match statement:
```rust
match NonNull::new(raw_ptr) {
Some(data) => data,
None => return Err(TryReserveError::new(layout.size())),
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031452174
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3578383620
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -371,6 +457,39 @@ impl MutableBuffer {
}
}
+ /// mirrors [`MutableBuffer::reallocate`] but returns
+ /// [`MutableBufferError::LayoutError`] instead of panicking on an invalid layout.
+ #[cold]
+ #[allow(dead_code)]
Review Comment:
```suggestion
#[expect(dead_code)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3583517428
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
This is still a panic on allocation failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5132567156
ah i more meant inside the existing infallible methods, for example in `with_capacity()` we could customize it to do `handle_alloc_error` on `AllocationError` and such
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5186598277
nice! thanks @Jefffrey Ill make a follow up PR to replace the infallible methods with these new `try_*()` methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5093583838
> just having a hard time determining if there are any performance regressions or its just noise in the benchmarks 🤔
nothing logically should have changed right? just a couple extra boolean checks for things like
```rust
let total_bytes = repeat_count
.checked_mul(bytes_per_copy)
.ok_or(MutableBufferError::LengthOverflow)?;
```
it is weird how `MutableBuffer::from_iter bool ` seem to regress over multiple benchmark calls
```
MutableBuffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.36 4.2±0.04ms ? ?/sec
```
&
```
MutableBuffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.37 4.2±0.03ms ? ?/sec
```
but then again `bench_bool/bench_bool ` seems to have also flipped
**old**
```
bench_bool/bench_bool 1.00 296.1±0.40µs 1688.8 MB/sec 1.22 361.2±0.21µs 1384.3 MB/sec
```
**new**
```
bench_bool/bench_bool 1.24 362.6±1.56µs 1379.1 MB/sec 1.00 293.1±0.32µs 1706.1 MB/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5086902199
just having a hard time determining if there are any performance regressions or its just noise in the benchmarks 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993569791
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-4993496337)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
bench_bool/bench_bool 1.00 296.1±0.40µs 1688.8 MB/sec 1.22 361.2±0.21µs 1384.3 MB/sec
bench_decimal128_builder 1.00 200.1±7.03µs ? ?/sec 1.00 200.6±7.54µs ? ?/sec
bench_decimal256_builder 1.00 201.4±7.60µs ? ?/sec 1.00 201.9±7.87µs ? ?/sec
bench_decimal32_builder 1.00 81.3±2.38µs ? ?/sec 1.00 81.0±2.52µs ? ?/sec
bench_decimal64_builder 1.01 99.8±4.51µs ? ?/sec 1.00 99.2±4.62µs ? ?/sec
bench_primitive/bench_primitive 1.00 48.7±0.28µs 80.3 GB/sec 1.00 48.7±0.12µs 80.2 GB/sec
bench_primitive/bench_string 1.00 5.5±0.04ms 1174.2 MB/sec 1.01 5.6±0.25ms 1161.2 MB/sec
bench_primitive_nulls/bench_primitive_nulls 1.03 1205.6±142.05µs ? ?/sec 1.00 1166.4±119.79µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 15.3 MiB |
| Avg memory | 8.6 MiB |
| CPU user | 72.9s |
| CPU sys | 6.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 14.9 MiB |
| Avg memory | 8.3 MiB |
| CPU user | 72.3s |
| CPU sys | 6.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056176370
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=1024 1.00 3.2±0.04µs ? ?/sec 1.00 3.2±0.04µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=3 1.00 93.5±0.49ns ? ?/sec 1.04 97.3±0.45ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=64 1.00 313.2±1.78ns ? ?/sec 1.05 330.1±2.20ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=100 n=8192 1.00 25.0±0.24µs ? ?/sec 1.02 25.4±0.18µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=1024 1.00 2.4±0.05µs ? ?/sec 1.00 2.4±0.01µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=3 1.12 78.4±0.28ns ? ?/sec 1.00 70.3±0.48ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=64 1.01 263.6±1.94ns ? ?/sec 1.00 259.9±0.78ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=20 n=8192 1.00 17.9±0.76µs ? ?/sec 1.03 18.3±0.04µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=1024 1.00 2.4±0.00µs ? ?/sec 1.00 2.4±0.00µs ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=3 1.03 72.4±0.26ns ? ?/sec 1.00 70.4±0.24ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=64 1.00 206.1±1.15ns ? ?/sec 1.10 225.8±0.22ns ? ?/sec
MutableBuffer repeat slice/extend_from_slice loop/slice_len=3 n=8192 1.00 17.2±1.01µs ? ?/sec 1.05 18.1±0.03µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=1024 1.01 1260.8±1.21ns ? ?/sec 1.00 1252.1±1.37ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=3 1.23 108.5±1.30ns ? ?/sec 1.00 88.2±0.61ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=64 1.00 210.5±1.12ns ? ?/sec 1.09 229.7±1.17ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=100 n=8192 1.00 10.5±0.00µs ? ?/sec 1.00 10.5±0.01µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=1024 1.00 364.8±0.95ns ? ?/sec 1.00 364.6±0.83ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=3 1.08 81.7±0.56ns ? ?/sec 1.00 75.5±1.10ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=64 1.01 153.9±0.79ns ? ?/sec 1.00 153.1±0.78ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=20 n=8192 1.00 2.0±0.00µs ? ?/sec 1.00 2.0±0.00µs ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=1024 1.01 185.4±1.13ns ? ?/sec 1.00 182.9±1.10ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=3 1.00 80.6±0.91ns ? ?/sec 1.01 81.6±1.03ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=64 1.02 106.3±1.46ns ? ?/sec 1.00 104.0±0.37ns ? ?/sec
MutableBuffer repeat slice/repeat_slice_n_times/slice_len=3 n=8192 1.03 417.8±1.13ns ? ?/sec 1.00 404.6±1.50ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 235.1s |
| Peak memory | 9.7 MiB |
| Avg memory | 5.2 MiB |
| CPU user | 228.1s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 235.1s |
| Peak memory | 8.6 MiB |
| Avg memory | 4.1 MiB |
| CPU user | 230.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3584089126
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
good catch. The one way this could fail is if we OOM.
We could do something like this
```suggestion
let raw_ptr = unsafe { std::alloc::alloc(layout) };
if raw_ptr.is_null() {
return Err(MutableBufferError::OOM);
}
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
```
But I think `handle_alloc_error` already deals with this. I think running out of memory is a condition where the program should panic.
We could bubble up the error to the caller but there aren't many things you can do when your process runs out of memory, I'm sure it would be similar handling to what `handle_alloc_error` already does. I don't have too strong of an opinion either way, happy to try either approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3584467438
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
My interest in this PR is to help with: https://github.com/apache/datafusion/issues/22758
The linux allocator famously does not limit allocations, and so you can get reaped by the OOM killer easily.
If we have a global allocator that internally limits how many allocations it dishes out, we can course correct in code if an error is returned. A panic still works, but this is a panic which is a pretty blunt instrument and means we can't take corrective action in code easily, which is what I found in testing.
I.e, if we have a memory limited allocator we can use this as a gauge to whether we need to spill memory to disk etc... so it's very useful as a tool there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3589455945
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -121,6 +146,12 @@ impl MutableBuffer {
Self::with_capacity(capacity)
}
+ /// Fallible version of [`MutableBuffer::new`].
+ #[inline]
+ pub fn try_new(capacity: usize) -> Result<Self, MutableBufferError> {
Review Comment:
i wonder if we should bother with having a `try_new` if it just delegates to `try_with_capacity` 🤔
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
are there any downstream concerns with replacing what we previously did (calling `handle_alloc_error()`) with just an error and relying on downstream to properly handle (in this case theyd just do a regular panic now)
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -313,74 +406,22 @@ impl MutableBuffer {
slice_to_repeat: &[T],
repeat_count: usize,
) {
- if repeat_count == 0 || slice_to_repeat.is_empty() {
- return;
- }
-
- let bytes_to_repeat = size_of_val(slice_to_repeat);
- let repeated_bytes = repeat_count
- .checked_mul(bytes_to_repeat)
- .expect("repeated slice byte length overflow");
- self.len
- .checked_add(repeated_bytes)
- .expect("mutable buffer length overflow");
-
- // Ensure capacity
- self.reserve(repeated_bytes);
-
- // Save the length before we do all the copies to know where to start from
- let length_before = self.len;
-
- // Copy the initial slice once so we can use doubling strategy on it
- self.extend_from_slice(slice_to_repeat);
-
- // This tracks how much bytes we have added by repeating so far
- let added_repeats_length = bytes_to_repeat;
- assert_eq!(
- self.len - length_before,
- added_repeats_length,
- "should copy exactly the same number of bytes"
- );
-
- // Number of times the slice was repeated
- let mut already_repeated_times = 1;
-
- // We will use doubling strategy to fill the buffer in log(repeat_count) steps
- while already_repeated_times < repeat_count {
- // How many slices can we copy in this iteration
- // (either double what we have, or just the remaining ones)
- let number_of_slices_to_copy =
- already_repeated_times.min(repeat_count - already_repeated_times);
- let number_of_bytes_to_copy = number_of_slices_to_copy * bytes_to_repeat;
-
- unsafe {
- // Get to the start of the data before we started copying anything
- let src = self.data.as_ptr().add(length_before) as *const u8;
-
- // Go to the current location to copy to (end of current data)
- let dst = self.data.as_ptr().add(self.len);
-
- // SAFETY: the pointers are not overlapping as there is `number_of_bytes_to_copy` or less between them
Review Comment:
we probably should preserve some of these comments; especially the safety one at the very least
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -129,24 +160,35 @@ impl MutableBuffer {
/// then `isize::MAX`, then this function will panic.
#[inline]
pub fn with_capacity(capacity: usize) -> Self {
- let capacity = bit_util::round_upto_multiple_of_64(capacity);
+ Self::try_with_capacity(capacity).unwrap_or_else(|e| panic!("{e}"))
+ }
+
+ /// Fallible version of [`MutableBuffer::with_capacity`].
+ #[inline]
+ pub fn try_with_capacity(capacity: usize) -> Result<Self, MutableBufferError> {
+ let capacity = capacity
+ .checked_next_multiple_of(64)
Review Comment:
`checked_next_multiple_of` is quite nice; i wonder if that means we can deprecate `round_upto_multiple_of_64` 🤔 (in a followup)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4983657438
>it would be good to preserve some of the useful comments as it looks like theyre getting lost in the port
Ill add the comments back in
> are there any downstream concerns with replacing what we previously did (calling handle_alloc_error()) with just an error and relying on downstream to properly handle
im not sure. I think the best person to ask would be @alamb.
from the docs for [handle_alloc_error](https://doc.rust-lang.org/beta/std/alloc/fn.handle_alloc_error.html)
> it may either panic (resulting in unwinding or aborting as per configuration for all panics), or abort the process (with no unwinding).
it seems that once this branch is hit there isn't much for the process to do anyway? in both cases it aborts the process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5052882061
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056131251
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
bench_bool/bench_bool 1.24 362.6±1.56µs 1379.1 MB/sec 1.00 293.1±0.32µs 1706.1 MB/sec
bench_decimal128_builder 1.00 200.8±7.41µs ? ?/sec 1.00 200.9±6.56µs ? ?/sec
bench_decimal256_builder 1.00 201.3±7.51µs ? ?/sec 1.00 202.1±6.89µs ? ?/sec
bench_decimal32_builder 1.00 81.4±2.28µs ? ?/sec 1.00 81.4±2.59µs ? ?/sec
bench_decimal64_builder 1.00 99.9±4.61µs ? ?/sec 1.00 100.2±4.93µs ? ?/sec
bench_primitive/bench_primitive 1.00 48.5±0.26µs 80.6 GB/sec 1.00 48.5±0.18µs 80.5 GB/sec
bench_primitive/bench_string 1.02 5.4±0.02ms 1207.9 MB/sec 1.00 5.3±0.02ms 1230.0 MB/sec
bench_primitive_nulls/bench_primitive_nulls 1.00 1205.3±140.54µs ? ?/sec 1.00 1200.9±145.49µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 18.1 MiB |
| Avg memory | 8.6 MiB |
| CPU user | 73.1s |
| CPU sys | 6.8s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 15.9 MiB |
| Avg memory | 8.6 MiB |
| CPU user | 71.8s |
| CPU sys | 6.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4958151356
> I just saw your PR when I raised my own one: #10329! Sorry for the double up of work...
@cetra3 No worries — to avoid having the maintainers review two separate PRs, would you mind closing #10329 and reviewing this one instead? That way we can consolidate our approaches and make it easier to review. We can also split off the follow-up work of propagating the fallible version of the API throughout the arrow codebase. Let me know if this makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5037331414
> Benchmark for [this request](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5031452174) failed.
>
> Last 20 lines of output:
>
> Click to expand
> ```
> record_batch
> regexp_kernels
> row_format
> row_group_index_reader
> row_selection_cursor
> row_selector
> serde
> sort_kernel
> string_dictionary_builder
> string_run_builder
> string_run_iterator
> substring_kernels
> take_kernels
> union_array
> variant_builder
> variant_kernels
> variant_validation
> view_types
> writer_overhead
> zip_kernels
> ```
>
> [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
@cetra3 this seems to have failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5050818655
> It uses an opaque struct rather than an enum, which means we are more free to adjust implementation without changing internals, making it a breaking api change.
it just came to mind that we can mark this enum as `#[non_exhaustive]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5052897606
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5052882061)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5052882061-1239-wdb7n 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing rich-T-kid/fallible-mutable-buffer-api (6f03becc7066aa8a7edfc5d24a472db537bd571f) to b19aa65 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b19aa6538e7b3eac3f177f5dde4c3bc3615a1d1c..6f03becc7066aa8a7edfc5d24a472db537bd571f)
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench sql_planner
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056137488
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10317#issuecomment-5056082827)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main rich-T-kid_fallible-mutable-buffer-api
----- ---- --------------------------------------
Buffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
MutableBuffer iter bitset 1.00 46.7±1.80ms ? ?/sec 1.00 46.6±0.55ms ? ?/sec
MutableBuffer::from_iter bool 1.00 3.1±0.01ms ? ?/sec 1.37 4.2±0.03ms ? ?/sec
from_slice 1.00 222.4±0.95µs ? ?/sec 1.15 255.2±0.70µs ? ?/sec
from_slice prepared 1.01 213.6±0.59µs ? ?/sec 1.00 211.5±0.57µs ? ?/sec
mutable 1.01 279.5±0.61µs ? ?/sec 1.00 275.5±0.84µs ? ?/sec
mutable extend 1.00 541.9±0.96µs ? ?/sec 1.01 549.6±0.85µs ? ?/sec
mutable iter extend_from_slice 1.01 1154.6±1.54µs ? ?/sec 1.00 1139.6±13.16µs ? ?/sec
mutable prepared 1.02 185.6±0.57µs ? ?/sec 1.00 182.5±0.81µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 95.0s |
| Peak memory | 141.2 MiB |
| Avg memory | 25.6 MiB |
| CPU user | 77.9s |
| CPU sys | 13.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 141.3 MiB |
| Avg memory | 25.4 MiB |
| CPU user | 80.0s |
| CPU sys | 14.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5026540092
could we run the benchmarks again to make sure this PR didnt cause a 36% regression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4954648431
I just saw your PR when I raised my own one: https://github.com/apache/arrow-rs/pull/10329! Sorry for the double up of work...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4939076069
> I made new methods instead of adding conditional checks/wrappers for each method because its easier to slowly roll these changes out to call sites instead of swapping.
once this is merged Ill work on updating call sites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3580348415
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -397,6 +421,13 @@ impl MutableBuffer {
reservation.resize(self.layout.size());
}
}
+ Ok(())
+ }
+ #[cold]
+ #[allow(dead_code)]
Review Comment:
setting this to #[expect(dead_code)] work fine for local development but it cause this ci to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3584124683
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -140,13 +174,13 @@ impl MutableBuffer {
NonNull::new(raw_ptr).unwrap_or_else(|| handle_alloc_error(layout))
Review Comment:
I use a memory limited global allocator in some scenarios *already*, and if we want to move to custom allocators, we don't want to capture the panic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-4963787444
Hey no worries I will close it off.
I do like the following differences from this PR:
* It doesn't duplicate the logic, i.e, the fallible versions are just `try_<method>().unwrap()` which means we don't need to adjust it in two places
* It uses an opaque struct rather than an enum, which means we are more free to adjust implementation without changing internals, making it a breaking api change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#discussion_r3589590919
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -129,24 +160,35 @@ impl MutableBuffer {
/// then `isize::MAX`, then this function will panic.
#[inline]
pub fn with_capacity(capacity: usize) -> Self {
- let capacity = bit_util::round_upto_multiple_of_64(capacity);
+ Self::try_with_capacity(capacity).unwrap_or_else(|e| panic!("{e}"))
+ }
+
+ /// Fallible version of [`MutableBuffer::with_capacity`].
+ #[inline]
+ pub fn try_with_capacity(capacity: usize) -> Result<Self, MutableBufferError> {
+ let capacity = capacity
+ .checked_next_multiple_of(64)
Review Comment:
created a follow up ticket : https://github.com/apache/arrow-rs/issues/10350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5132454956
after this is merged ill make a PR to wire up call sites to use the new `try_*()` methods. Ideally we can propogate these errors up the call stack as https://docs.rs/arrow/latest/arrow/error/enum.ArrowError.html#variant.MemoryError
> we could technically match on the errors and if we hit AllocationError just call handle_alloc_error as we essentially did before
im not opposed to this but is there any reason why we shouldnt just pass up OOM errors if possible instead of panicking?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat/chore: introduce fallible alternatives for MutableBuffer [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10317:
URL: https://github.com/apache/arrow-rs/pull/10317#issuecomment-5185933937
thanks @Rich-T-kid, @cetra3 & @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org