You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "cetra3 (via GitHub)" <gi...@apache.org> on 2026/10/11 15:32:22 UTC
[PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
cetra3 opened a new pull request, #11511:
URL: https://github.com/apache/arrow-rs/pull/11511
# Which issue does this PR close?
- N/A
This PR supersedes #11505 and #11506. It makes the same changes in a general form.
# Rationale for this change
Every access to a variant value pays for checks that the access does not use. `Variant::new` and the unvalidated constructors of `VariantMetadata`, `VariantObject` and `VariantList` read the first and last offsets of each header and trim the byte slices. A field lookup then validates the UTF-8 of every dictionary name that its binary search probes. Kernels such as `variant_get` construct a full `Variant` for every row, and then walk the path through it.
For most kernel work, a value needs much less: the header of each object or list along a path, the names that the binary search probes, and the bytes of the value that it finds.
# What changes are included in this PR?
- A new module, `parquet_variant::view` (`#[doc(hidden)]`), adds `MetadataView`, `ValueView`, `ObjectView` and `ListView`.
- Building a view reads one header and nothing else.
- Each accessor reads and bounds-checks only the bytes that it needs, and returns an error on malformed input. It never panics.
- Field names are compared as bytes, which orders them as `str` does, so a search does not validate UTF-8.
- A view promotes to the existing types: `MetadataView::into_metadata`, `ObjectView::into_object`, `ListView::into_list` and `ValueView::to_variant` add the first and last offset checks that unvalidated instances guarantee.
- `VariantMetadata`, `VariantObject` and `VariantList` are now built by promoting a view, and their accessors (`get`, `field`, `field_name`, `iter`, `get_entry`) delegate to it. Their public behavior is unchanged.
- `variant_get` reads unshredded columns through the views, for field and index paths and for typed and `Variant` output. A row reads only the headers and names along the path, and decodes only the value that it finds.
- `shred_variant` and `unshred_variant` copy leftover object fields by field id with their encoded bytes, through a new `#[doc(hidden)]` `ObjectBuilder::insert_encoded`. A leftover field is no longer decoded into a `Variant` and then re-encoded.
- `ObjectBuilder::finish` sorts fields with a new `MetadataBuilder::cmp_field_names`. `ReadOnlyMetadataBuilder` compares the names as bytes.
- `VariantMetadata::borrowed_field_id` is removed. No caller in the repository needs it after the changes above. Its tests now cover `get_entry`.
# Are these changes tested?
The existing tests of `parquet-variant`, `parquet-variant-compute`, `parquet-variant-json`, and the `parquet` variant integration tests pass. The public types now delegate to the views, so their tests also cover the views.
Benchmarks (local run, `main` vs this PR, back to back, Apple Silicon laptop):
| benchmark | main | this PR | change |
|---|---:|---:|---:|
| `variant_get_unshredded_object_path_262k_rows` | 21.93 ms | 11.98 ms | −45% |
| `variant_get_list_index_unshredded` | 573.4 µs | 266.5 µs | −54% |
| `variant_get_utf8_from_int` | 272.4 µs | 172.8 µs | −37% |
| `variant_get_utf8_from_unshredded_string` | 336.0 µs | 226.5 µs | −33% |
| `variant_get_binary_from_unshredded_binary` | 283.0 µs | 179.2 µs | −37% |
| `variant_get_utf8_from_list` | 364.6 µs | 329.3 µs | −10% |
| `variant_get_utf8_from_map_in_list` | 376.4 µs | 331.1 µs | −12% |
| `shred_variant_partial_object_8k_rows` | 14.03 ms | 9.29 ms | −34% |
| `shred_variant_unmatched_object_8k_rows` | 18.50 ms | 11.65 ms | −37% |
| `variant_get_list_index_shredded` | 59.4 µs | 58.8 µs | −1% |
| `variant_get_shredded_utf8` | 343.1 ns | 343.6 ns | +0% |
| `object_get_by_name/128` | 86.0 ns | 69.1 ns | −20% |
| `object_iter/128` | 2.0 µs | 1.6 µs | −21% |
| `object_field_name/128` | 1.1 µs | 962.6 ns | −12% |
| `metadata_iter/128` | 688.5 ns | 609.8 ns | −11% |
| `bench_validate_large_object` | 19.39 ms | 19.46 ms | +0% |
| `bench_validate_large_nested_list` | 7.18 ms | 7.34 ms | +2% |
The 16 `variant_get_decimal_*` benchmarks are 25–36% faster. No benchmark in `variant_kernels` or `variant_validation` is more than 2% slower.
On real-world telemetry data with per-row metadata, `variant_get` on unshredded object fields is 1.6–1.9x faster than `main` for keys that most rows contain, and 2.2–2.4x faster for keys that are rare or absent.
# Are there any significant user-facing changes?
No public API changes, other than the new `#[doc(hidden)]` items. Behavior differences:
- On malformed variant bytes, `variant_get`, `shred_variant` and the residual copy in `unshred_variant` return an error, or a null under safe casting. Before, these paths could panic.
- Some out-of-range errors from metadata and list accessors have new message text.
- A caller outside the repository that passes field names borrowed from the metadata dictionary to `ObjectBuilder::insert_bytes` now resolves them through `get_entry` and the existing name cache, not through `borrowed_field_id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110799378
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635038)
**Target:** variant_kernels
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
resources:
arch: amd64
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
shred_variant_partial_object_8k_rows 1.30 14.9±0.27ms ? ?/sec 1.00 11.4±0.20ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.36 18.9±0.36ms ? ?/sec 1.00 13.8±0.25ms ? ?/sec
variant_get_binary_from_string 1.28 401.5±2.54µs ? ?/sec 1.00 313.6±2.93µs ? ?/sec
variant_get_binary_from_unshredded_binary 1.08 357.5±6.11µs ? ?/sec 1.00 329.5±3.97µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_non_finite_8192_safe 1.17 413.1±6.19µs ? ?/sec 1.00 353.7±2.17µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_safe 1.06 411.6±10.51µs ? ?/sec 1.00 389.0±3.31µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_strict 1.30 507.4±13.49µs ? ?/sec 1.00 389.2±2.48µs ? ?/sec
variant_get_decimal_double_to_decimal256(40,-2)_scale_down_8192_safe 1.07 697.4±12.25µs ? ?/sec 1.00 651.6±5.59µs ? ?/sec
variant_get_decimal_double_to_decimal64(3,0)_precision_overflow_8192_safe 1.07 357.7±4.65µs ? ?/sec 1.00 334.7±4.68µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_safe 1.33 371.6±5.27µs ? ?/sec 1.00 280.1±2.21µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_strict 1.47 412.1±16.36µs ? ?/sec 1.00 280.1±2.83µs ? ?/sec
variant_get_decimal_float_to_decimal64(3,0)_precision_overflow_8192_safe 1.32 370.5±5.74µs ? ?/sec 1.00 281.1±3.66µs ? ?/sec
variant_get_decimal_int32_to_decimal32(8,0)_precision_overflow_8192_safe 1.40 416.2±5.88µs ? ?/sec 1.00 298.2±2.90µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_safe 1.34 399.8±8.51µs ? ?/sec 1.00 299.0±2.65µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_strict 1.41 421.3±6.23µs ? ?/sec 1.00 298.5±3.08µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_safe 1.17 404.9±6.02µs ? ?/sec 1.00 344.7±2.50µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_strict 1.34 460.6±6.51µs ? ?/sec 1.00 345.0±2.08µs ? ?/sec
variant_get_decimal_int64_to_decimal256(76,-1)_scale_down_8192_safe 1.10 455.2±7.88µs ? ?/sec 1.00 413.4±2.10µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-1)_scale_before_narrowing_8192_safe 1.18 392.8±5.63µs ? ?/sec 1.00 332.3±2.59µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-20)_all_zero_8192_safe 1.30 396.4±6.68µs ? ?/sec 1.00 304.6±2.89µs ? ?/sec
variant_get_list_index_all_oob_int64_64_rows 1.00 1228.4±42.09ns ? ?/sec 1.03 1271.2±161.67ns ? ?/sec
variant_get_list_index_shredded 1.00 39.2±0.93µs ? ?/sec 1.01 39.8±0.99µs ? ?/sec
variant_get_list_index_unshredded 1.74 695.9±12.24µs ? ?/sec 1.00 399.2±8.02µs ? ?/sec
variant_get_primitive 1.00 400.6±8.40ns ? ?/sec 1.00 401.8±9.62ns ? ?/sec
variant_get_shredded_utf8 1.00 488.6±5.69ns ? ?/sec 1.01 495.0±8.41ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.28 28.1±0.46ms ? ?/sec 1.00 21.9±0.49ms ? ?/sec
variant_get_utf8_from_date 1.26 405.9±2.09µs ? ?/sec 1.00 322.8±5.05µs ? ?/sec
variant_get_utf8_from_decimal 1.18 360.1±4.93µs ? ?/sec 1.00 305.6±5.56µs ? ?/sec
variant_get_utf8_from_float 1.18 344.4±5.00µs ? ?/sec 1.00 292.4±5.84µs ? ?/sec
variant_get_utf8_from_int 1.18 343.0±5.16µs ? ?/sec 1.00 291.3±4.97µs ? ?/sec
variant_get_utf8_from_list 1.00 433.3±9.00µs ? ?/sec 1.15 499.0±10.94µs ? ?/sec
variant_get_utf8_from_map_in_list 1.00 436.9±12.51µs ? ?/sec 1.12 490.3±7.26µs ? ?/sec
variant_get_utf8_from_time 1.21 594.0±24.81µs ? ?/sec 1.00 489.4±8.26µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.18 421.9±6.48µs ? ?/sec 1.00 358.6±7.62µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.17 421.0±5.58µs ? ?/sec 1.00 359.4±7.23µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.15 411.4±8.01µs ? ?/sec 1.00 358.9±5.84µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-9vkbev77-575f51ab-dm7t
**Instance:** c4-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6110635038-3296-qrk5b 6.12.94+ #1 SMP Fri Aug 21 07:56:42 UTC 2026 x86_64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.1s |
| Peak memory | 350.2 MiB |
| Avg memory | 270.3 MiB |
| CPU user | 368.1s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 395.1s |
| Peak memory | 350.3 MiB |
| Avg memory | 276.8 MiB |
| CPU user | 390.3s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110635038:finish:3028ea642cd2caebcf83ce100389d319 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110636254
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110634871)
**Target:** variant_kernels
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110634871:start:dca9baee4e3e8e47f937c4ad1dc0d7f6 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256786133
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "Jefffrey (via GitHub) via github" <gi...@arrow.apache.org>.
Jefffrey commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6840793421
@sdf-jkl would you have capacity to help review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111166083
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111083003)
**Target:** variant_validation
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
bench_validate_complex_object 1.00 181.9±2.41µs ? ?/sec 1.00 181.9±2.40µs ? ?/sec
bench_validate_large_nested_list 1.01 20.2±0.45ms ? ?/sec 1.00 20.1±0.43ms ? ?/sec
bench_validate_large_object 1.01 55.6±2.19ms ? ?/sec 1.00 55.1±1.95ms ? ?/sec
metadata_iter/128 1.08 1003.2±73.48ns 121.7 MElem/sec 1.00 932.2±12.99ns 130.9 MElem/sec
metadata_iter/32 1.08 256.3±17.54ns 119.1 MElem/sec 1.00 236.7±3.08ns 128.9 MElem/sec
metadata_iter/8 1.07 61.6±3.54ns 123.8 MElem/sec 1.00 57.6±0.88ns 132.5 MElem/sec
object_field_name/128 1.07 1183.2±35.18ns 103.2 MElem/sec 1.00 1109.5±5.02ns 110.0 MElem/sec
object_field_name/32 1.06 301.0±7.40ns 101.4 MElem/sec 1.00 283.5±1.41ns 107.7 MElem/sec
object_field_name/8 1.07 74.2±2.07ns 102.8 MElem/sec 1.00 69.4±0.43ns 109.9 MElem/sec
object_get_by_name/128 1.11 110.8±1.06ns ? ?/sec 1.00 99.8±0.97ns ? ?/sec
object_get_by_name/32 1.04 72.7±0.88ns ? ?/sec 1.00 70.2±0.20ns ? ?/sec
object_get_by_name/8 1.00 59.6±0.80ns ? ?/sec 1.01 60.1±0.46ns ? ?/sec
object_iter/128 1.11 2.6±0.05µs 46.5 MElem/sec 1.00 2.4±0.02µs 51.5 MElem/sec
object_iter/32 1.12 672.6±10.73ns 45.4 MElem/sec 1.00 598.0±5.59ns 51.0 MElem/sec
object_iter/8 1.13 169.4±2.15ns 45.0 MElem/sec 1.00 149.4±1.51ns 51.1 MElem/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_validation — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-ufagysz2-36d0891d-l4j2
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6111083003-3300-ppll7 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 151.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 150.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.5 MiB |
| CPU user | 144.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111083003:finish:f485359e72bb5cf9890706fcd0b5d992 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111108165
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106927)
**Target:** variant_kernels
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
resources:
arch: amd64
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111106927:start:6cdc77e97bbb679031ccbddbe27d4355 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635211
run benchmark variant_validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106677
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111107480
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106677)
**Target:** variant_kernels
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111106677:start:6e9cd5ce31ad8960432bb052ee7f3f20 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256993002
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256786133)
**Target:** variant_kernels
Comparing variant-typestate-views (c5c21219423b575cfa76bbea6bf7c3ade5290de5) to 33d338d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/33d338ddf661b6e85364a9b0bd4d367c3468257d..c5c21219423b575cfa76bbea6bf7c3ade5290de5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
shred_variant_partial_object_8k_rows 1.44 21.3±0.08ms ? ?/sec 1.00 14.7±0.06ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.50 26.7±0.14ms ? ?/sec 1.00 17.7±0.08ms ? ?/sec
variant_get_binary_from_string 1.27 478.5±0.78µs ? ?/sec 1.00 378.2±12.02µs ? ?/sec
variant_get_binary_from_unshredded_binary 1.27 464.4±2.64µs ? ?/sec 1.00 365.7±14.56µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_non_finite_8192_safe 1.24 506.3±1.52µs ? ?/sec 1.00 409.7±12.40µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_safe 1.31 530.7±19.13µs ? ?/sec 1.00 406.6±10.81µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_strict 1.37 556.9±23.88µs ? ?/sec 1.00 406.4±10.79µs ? ?/sec
variant_get_decimal_double_to_decimal256(40,-2)_scale_down_8192_safe 1.17 1052.1±69.72µs ? ?/sec 1.00 897.6±1.60µs ? ?/sec
variant_get_decimal_double_to_decimal64(3,0)_precision_overflow_8192_safe 1.27 503.3±1.75µs ? ?/sec 1.00 397.6±12.95µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_safe 1.30 471.9±1.86µs ? ?/sec 1.00 363.3±13.35µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_strict 1.41 511.5±1.35µs ? ?/sec 1.00 363.1±13.52µs ? ?/sec
variant_get_decimal_float_to_decimal64(3,0)_precision_overflow_8192_safe 1.25 516.5±2.16µs ? ?/sec 1.00 414.8±13.64µs ? ?/sec
variant_get_decimal_int32_to_decimal32(8,0)_precision_overflow_8192_safe 1.27 479.4±1.44µs ? ?/sec 1.00 376.1±12.24µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_safe 1.29 466.5±1.22µs ? ?/sec 1.00 360.2±12.60µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_strict 1.41 508.2±2.06µs ? ?/sec 1.00 360.6±12.95µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_safe 1.29 523.6±29.69µs ? ?/sec 1.00 405.3±3.38µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_strict 1.41 572.8±39.24µs ? ?/sec 1.00 405.6±3.08µs ? ?/sec
variant_get_decimal_int64_to_decimal256(76,-1)_scale_down_8192_safe 1.25 537.9±30.88µs ? ?/sec 1.00 431.2±6.58µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-1)_scale_before_narrowing_8192_safe 1.28 479.5±1.05µs ? ?/sec 1.00 373.2±11.65µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-20)_all_zero_8192_safe 1.35 475.1±19.86µs ? ?/sec 1.00 352.8±4.65µs ? ?/sec
variant_get_decimal_short_string_to_decimal256(40,-2)_scale_down_8192_safe 1.20 575.4±2.07µs ? ?/sec 1.00 479.7±12.39µs ? ?/sec
variant_get_decimal_short_string_to_decimal256(76,2)_wide_8192_safe 1.11 957.1±3.08µs ? ?/sec 1.00 858.7±11.58µs ? ?/sec
variant_get_decimal_short_string_to_decimal32(7,2)_valid_8192_safe 1.27 549.2±2.11µs ? ?/sec 1.00 432.7±9.81µs ? ?/sec
variant_get_decimal_short_string_to_decimal32(7,2)_valid_8192_strict 1.38 598.7±7.10µs ? ?/sec 1.00 432.5±9.83µs ? ?/sec
variant_get_decimal_string_to_decimal128(38,3)_long_8192_safe 1.08 1106.7±7.65µs ? ?/sec 1.00 1020.2±10.40µs ? ?/sec
variant_get_decimal_string_to_decimal128(38,3)_long_8192_strict 1.16 1180.2±13.64µs ? ?/sec 1.00 1017.7±10.80µs ? ?/sec
variant_get_list_index_all_oob_int64_64_rows 1.00 1784.2±21.55ns ? ?/sec 1.00 1781.4±22.51ns ? ?/sec
variant_get_list_index_shredded 1.00 45.7±1.54µs ? ?/sec 1.01 46.2±1.34µs ? ?/sec
variant_get_list_index_unshredded 1.41 1008.8±13.37µs ? ?/sec 1.00 715.6±22.93µs ? ?/sec
variant_get_primitive 1.00 561.6±5.49ns ? ?/sec 1.01 566.1±5.73ns ? ?/sec
variant_get_shredded_utf8 1.01 718.1±4.88ns ? ?/sec 1.00 710.9±7.22ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.05 37.7±0.64ms ? ?/sec 1.00 36.0±1.06ms ? ?/sec
variant_get_utf8_from_date 1.27 568.3±5.10µs ? ?/sec 1.00 446.0±10.36µs ? ?/sec
variant_get_utf8_from_decimal 1.25 472.5±6.64µs ? ?/sec 1.00 376.9±18.58µs ? ?/sec
variant_get_utf8_from_float 1.31 446.9±1.10µs ? ?/sec 1.00 341.2±16.44µs ? ?/sec
variant_get_utf8_from_int 1.31 446.9±1.05µs ? ?/sec 1.00 340.7±14.36µs ? ?/sec
variant_get_utf8_from_list 1.00 657.2±10.69µs ? ?/sec 1.08 709.6±21.31µs ? ?/sec
variant_get_utf8_from_map_in_list 1.00 677.8±13.16µs ? ?/sec 1.08 733.2±19.15µs ? ?/sec
variant_get_utf8_from_time 1.18 777.1±2.73µs ? ?/sec 1.00 657.0±13.30µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.29 569.6±3.00µs ? ?/sec 1.00 442.5±15.03µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.29 569.7±3.06µs ? ?/sec 1.00 442.9±15.55µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.24 493.7±1.35µs ? ?/sec 1.00 396.6±16.84µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1o54riqp-92009ed2-g7jn
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6256786133-3576-l29vf 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 450.1s |
| Peak memory | 349.1 MiB |
| Avg memory | 285.6 MiB |
| CPU user | 447.0s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 455.1s |
| Peak memory | 349.0 MiB |
| Avg memory | 284.5 MiB |
| CPU user | 451.4s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6256786133:finish:f42ef3805fa4ea1863072247b35e9360 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635597
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635038)
**Target:** variant_kernels
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
resources:
arch: amd64
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110635038:start:6efdc8989cb3e07baaaf1232eb3f258b -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110634871
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111274624
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106927)
**Target:** variant_kernels
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
resources:
arch: amd64
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
shred_variant_partial_object_8k_rows 1.31 15.4±0.21ms ? ?/sec 1.00 11.8±0.32ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.38 19.8±0.32ms ? ?/sec 1.00 14.4±0.50ms ? ?/sec
variant_get_binary_from_string 1.40 419.4±5.08µs ? ?/sec 1.00 298.8±22.26µs ? ?/sec
variant_get_binary_from_unshredded_binary 1.16 359.8±7.65µs ? ?/sec 1.00 311.2±24.62µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_non_finite_8192_safe 1.19 420.3±3.28µs ? ?/sec 1.00 352.5±19.79µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_safe 1.11 405.0±3.63µs ? ?/sec 1.00 366.2±3.37µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_strict 1.37 501.0±6.86µs ? ?/sec 1.00 366.4±3.40µs ? ?/sec
variant_get_decimal_double_to_decimal256(40,-2)_scale_down_8192_safe 1.07 685.3±13.42µs ? ?/sec 1.00 641.4±6.26µs ? ?/sec
variant_get_decimal_double_to_decimal64(3,0)_precision_overflow_8192_safe 1.10 359.9±4.86µs ? ?/sec 1.00 328.5±20.57µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_safe 1.36 374.7±3.95µs ? ?/sec 1.00 275.0±20.35µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_strict 1.49 408.2±5.82µs ? ?/sec 1.00 273.7±20.17µs ? ?/sec
variant_get_decimal_float_to_decimal64(3,0)_precision_overflow_8192_safe 1.34 369.5±2.14µs ? ?/sec 1.00 276.0±20.59µs ? ?/sec
variant_get_decimal_int32_to_decimal32(8,0)_precision_overflow_8192_safe 1.44 420.4±7.31µs ? ?/sec 1.00 292.8±19.02µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_safe 1.41 410.6±6.04µs ? ?/sec 1.00 291.7±19.52µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_strict 1.41 415.5±4.70µs ? ?/sec 1.00 294.6±19.21µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_safe 1.25 403.3±6.72µs ? ?/sec 1.00 323.6±3.95µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_strict 1.41 454.3±6.42µs ? ?/sec 1.00 321.7±3.90µs ? ?/sec
variant_get_decimal_int64_to_decimal256(76,-1)_scale_down_8192_safe 1.16 454.9±7.06µs ? ?/sec 1.00 391.3±3.08µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-1)_scale_before_narrowing_8192_safe 1.26 409.8±5.77µs ? ?/sec 1.00 325.6±18.58µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-20)_all_zero_8192_safe 1.47 415.9±7.71µs ? ?/sec 1.00 282.7±5.69µs ? ?/sec
variant_get_list_index_all_oob_int64_64_rows 1.00 1237.3±45.58ns ? ?/sec 1.01 1245.1±89.02ns ? ?/sec
variant_get_list_index_shredded 1.03 39.4±0.44µs ? ?/sec 1.00 38.3±0.64µs ? ?/sec
variant_get_list_index_unshredded 1.74 698.1±16.67µs ? ?/sec 1.00 401.0±15.02µs ? ?/sec
variant_get_primitive 1.01 399.4±7.31ns ? ?/sec 1.00 396.0±4.96ns ? ?/sec
variant_get_shredded_utf8 1.00 491.0±4.81ns ? ?/sec 1.00 489.5±6.63ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.31 28.3±0.35ms ? ?/sec 1.00 21.7±0.72ms ? ?/sec
variant_get_utf8_from_date 1.41 435.5±5.10µs ? ?/sec 1.00 308.6±12.10µs ? ?/sec
variant_get_utf8_from_decimal 1.34 379.9±3.52µs ? ?/sec 1.00 284.4±23.08µs ? ?/sec
variant_get_utf8_from_float 1.31 354.7±3.32µs ? ?/sec 1.00 271.3±23.66µs ? ?/sec
variant_get_utf8_from_int 1.31 358.7±3.02µs ? ?/sec 1.00 272.9±22.91µs ? ?/sec
variant_get_utf8_from_list 1.00 441.5±8.75µs ? ?/sec 1.08 475.1±23.05µs ? ?/sec
variant_get_utf8_from_map_in_list 1.00 442.9±7.35µs ? ?/sec 1.08 476.4±22.04µs ? ?/sec
variant_get_utf8_from_time 1.31 620.1±7.25µs ? ?/sec 1.00 473.4±15.66µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.29 423.8±4.82µs ? ?/sec 1.00 328.3±21.04µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.27 424.8±6.14µs ? ?/sec 1.00 334.6±23.35µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.29 427.6±6.26µs ? ?/sec 1.00 332.7±22.55µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-137sax4c-d5ffafd9-t2fz
**Instance:** c4-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6111106927-3302-5gl8t 6.12.94+ #1 SMP Fri Aug 21 07:56:42 UTC 2026 x86_64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 2
BogoMIPS: 4600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 384 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 16 MiB (8 instances)
L3 cache: 260 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.1s |
| Peak memory | 350.4 MiB |
| Avg memory | 272.3 MiB |
| CPU user | 371.1s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 385.1s |
| Peak memory | 410.3 MiB |
| Avg memory | 276.3 MiB |
| CPU user | 381.2s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111106927:finish:c041cd6d1af8a4ff69531c68eb93ad72 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256783898
Hi @cetra3, your benchmark configuration could not be parsed (https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256783614).
**Error:** `invalid configuration: unknown field `BENCH_FILTER`, expected one of `env`, `baseline`, `changed`, `resources`, `shards` at line 3 column 1`
Usage:
```
run benchmark <name> # run specific benchmark(s)
run benchmarks # run default suite
run benchmarks <name1> <name2> # run specific benchmarks
```
Any benchmark name is accepted: `bench.sh` suite names (e.g. `tpch`, `clickbench_partitioned`, `wide_schema`) and Criterion bench targets (e.g. `sql_planner`) are resolved automatically. A name that matches neither fails on the runner.
Per-side configuration (`run benchmark tpch` followed by):
```yaml
env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G
```
Pod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal):
```yaml
resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64
```
Independent benchmark shards (optional; 1–8, default 1):
```yaml
run benchmark arrow_writer
shards: 4
```
Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111083815
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111083003)
**Target:** variant_validation
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111083003:start:4b071c34c609dcb59658604f9a445441 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "sdf-jkl (via GitHub) via github" <gi...@arrow.apache.org>.
sdf-jkl commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6841023569
Thanks @Jefffrey. It's been on my backlog for a few weeks. I'll prioritize it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110815840
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110634871)
**Target:** variant_kernels
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
shred_variant_partial_object_8k_rows 1.43 21.0±0.10ms ? ?/sec 1.00 14.7±0.07ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.46 26.2±0.15ms ? ?/sec 1.00 18.0±0.07ms ? ?/sec
variant_get_binary_from_string 1.14 481.4±0.83µs ? ?/sec 1.00 423.2±8.41µs ? ?/sec
variant_get_binary_from_unshredded_binary 1.07 471.2±4.38µs ? ?/sec 1.00 438.7±20.25µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_non_finite_8192_safe 1.08 497.4±1.52µs ? ?/sec 1.00 460.5±3.54µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_safe 1.10 514.8±1.69µs ? ?/sec 1.00 470.1±6.00µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_strict 1.16 540.1±2.45µs ? ?/sec 1.00 467.3±5.26µs ? ?/sec
variant_get_decimal_double_to_decimal256(40,-2)_scale_down_8192_safe 1.00 909.6±2.93µs ? ?/sec 1.01 919.3±3.41µs ? ?/sec
variant_get_decimal_double_to_decimal64(3,0)_precision_overflow_8192_safe 1.09 438.4±1.22µs ? ?/sec 1.00 403.1±7.85µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_safe 1.09 455.8±1.76µs ? ?/sec 1.00 419.2±8.94µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_strict 1.23 514.2±4.64µs ? ?/sec 1.00 417.7±9.03µs ? ?/sec
variant_get_decimal_float_to_decimal64(3,0)_precision_overflow_8192_safe 1.08 450.2±1.43µs ? ?/sec 1.00 415.0±6.83µs ? ?/sec
variant_get_decimal_int32_to_decimal32(8,0)_precision_overflow_8192_safe 1.20 492.4±4.23µs ? ?/sec 1.00 410.3±5.32µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_safe 1.18 468.8±1.45µs ? ?/sec 1.00 398.2±3.87µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_strict 1.28 508.9±1.58µs ? ?/sec 1.00 397.5±4.01µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_safe 1.16 516.1±2.01µs ? ?/sec 1.00 446.4±4.53µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_strict 1.25 559.4±3.57µs ? ?/sec 1.00 446.4±5.43µs ? ?/sec
variant_get_decimal_int64_to_decimal256(76,-1)_scale_down_8192_safe 1.01 529.4±3.47µs ? ?/sec 1.00 524.7±4.91µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-1)_scale_before_narrowing_8192_safe 1.17 478.0±0.90µs ? ?/sec 1.00 408.1±2.40µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-20)_all_zero_8192_safe 1.23 486.3±0.62µs ? ?/sec 1.00 394.5±2.70µs ? ?/sec
variant_get_list_index_all_oob_int64_64_rows 1.00 1776.5±20.73ns ? ?/sec 1.00 1783.3±19.66ns ? ?/sec
variant_get_list_index_shredded 1.00 45.2±1.51µs ? ?/sec 1.00 45.0±1.29µs ? ?/sec
variant_get_list_index_unshredded 1.42 1017.0±18.67µs ? ?/sec 1.00 715.9±2.51µs ? ?/sec
variant_get_primitive 1.02 571.7±13.69ns ? ?/sec 1.00 561.9±5.90ns ? ?/sec
variant_get_shredded_utf8 1.00 715.4±4.33ns ? ?/sec 1.00 717.0±6.00ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.09 37.3±0.58ms ? ?/sec 1.00 34.2±0.35ms ? ?/sec
variant_get_utf8_from_date 1.22 572.5±4.91µs ? ?/sec 1.00 469.3±3.91µs ? ?/sec
variant_get_utf8_from_decimal 1.09 473.2±6.75µs ? ?/sec 1.00 433.2±23.20µs ? ?/sec
variant_get_utf8_from_float 1.14 445.6±0.89µs ? ?/sec 1.00 391.1±9.38µs ? ?/sec
variant_get_utf8_from_int 1.15 446.2±1.04µs ? ?/sec 1.00 386.9±5.66µs ? ?/sec
variant_get_utf8_from_list 1.00 654.6±11.32µs ? ?/sec 1.26 826.3±3.15µs ? ?/sec
variant_get_utf8_from_map_in_list 1.00 667.5±12.23µs ? ?/sec 1.29 859.7±3.48µs ? ?/sec
variant_get_utf8_from_time 1.16 776.5±3.15µs ? ?/sec 1.00 671.2±3.59µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.25 572.0±2.89µs ? ?/sec 1.00 458.7±4.69µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.25 572.3±2.84µs ? ?/sec 1.00 458.7±4.68µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.13 493.6±0.99µs ? ?/sec 1.00 435.3±12.45µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-gidh403n-a9885889-hsc8
**Instance:** c4a-highcpu-48 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6110634871-3297-qh5h6 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 3 MiB (48 instances)
L1i cache: 3 MiB (48 instances)
L2 cache: 96 MiB (48 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 390.1s |
| Peak memory | 348.8 MiB |
| Avg memory | 272.9 MiB |
| CPU user | 384.0s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 349.0 MiB |
| Avg memory | 273.2 MiB |
| CPU user | 377.3s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110634871:finish:6f1b729f5763ecf53baa33f79a6c1690 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106927
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
resources:
arch: amd64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110636887
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635211)
**Target:** variant_validation
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110635211:start:141b80656125738acee287b36b72b9e3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256783614
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111270208
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111106677)
**Target:** variant_kernels
Comparing variant-typestate-views (b7e7a0ae35f2abadefae882975ab452c1f8f4920) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..b7e7a0ae35f2abadefae882975ab452c1f8f4920)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
shred_variant_partial_object_8k_rows 1.43 21.0±0.07ms ? ?/sec 1.00 14.7±0.08ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.47 26.2±0.17ms ? ?/sec 1.00 17.8±0.06ms ? ?/sec
variant_get_binary_from_string 1.26 482.3±0.76µs ? ?/sec 1.00 383.9±24.90µs ? ?/sec
variant_get_binary_from_unshredded_binary 1.27 462.0±3.51µs ? ?/sec 1.00 363.1±24.07µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_non_finite_8192_safe 1.32 497.4±1.20µs ? ?/sec 1.00 378.1±25.77µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_safe 1.31 512.2±1.60µs ? ?/sec 1.00 390.3±7.43µs ? ?/sec
variant_get_decimal_double_to_decimal128(20,3)_valid_8192_strict 1.38 537.3±1.51µs ? ?/sec 1.00 390.7±7.57µs ? ?/sec
variant_get_decimal_double_to_decimal256(40,-2)_scale_down_8192_safe 1.11 922.6±4.78µs ? ?/sec 1.00 829.6±13.32µs ? ?/sec
variant_get_decimal_double_to_decimal64(3,0)_precision_overflow_8192_safe 1.30 442.7±1.05µs ? ?/sec 1.00 340.8±21.49µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_safe 1.29 456.4±1.49µs ? ?/sec 1.00 354.2±20.99µs ? ?/sec
variant_get_decimal_float_to_decimal32(7,2)_valid_8192_strict 1.41 500.3±2.26µs ? ?/sec 1.00 353.9±20.70µs ? ?/sec
variant_get_decimal_float_to_decimal64(3,0)_precision_overflow_8192_safe 1.27 450.7±1.40µs ? ?/sec 1.00 355.9±21.19µs ? ?/sec
variant_get_decimal_int32_to_decimal32(8,0)_precision_overflow_8192_safe 1.28 481.4±1.23µs ? ?/sec 1.00 377.5±22.82µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_safe 1.30 468.6±1.17µs ? ?/sec 1.00 359.5±22.48µs ? ?/sec
variant_get_decimal_int32_to_decimal32(9,0)_valid_8192_strict 1.42 508.5±2.12µs ? ?/sec 1.00 359.2±22.22µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_safe 1.34 518.4±1.58µs ? ?/sec 1.00 387.9±2.79µs ? ?/sec
variant_get_decimal_int64_to_decimal128(38,2)_scale_up_8192_strict 1.44 560.0±3.39µs ? ?/sec 1.00 388.0±2.78µs ? ?/sec
variant_get_decimal_int64_to_decimal256(76,-1)_scale_down_8192_safe 1.23 529.6±1.32µs ? ?/sec 1.00 429.3±7.00µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-1)_scale_before_narrowing_8192_safe 1.28 479.5±0.92µs ? ?/sec 1.00 374.4±22.60µs ? ?/sec
variant_get_decimal_int64_to_decimal32(9,-20)_all_zero_8192_safe 1.39 488.4±0.72µs ? ?/sec 1.00 352.5±4.02µs ? ?/sec
variant_get_list_index_all_oob_int64_64_rows 1.00 1759.9±17.71ns ? ?/sec 1.01 1778.6±21.20ns ? ?/sec
variant_get_list_index_shredded 1.00 44.6±1.30µs ? ?/sec 1.03 46.0±1.13µs ? ?/sec
variant_get_list_index_unshredded 1.42 1013.1±14.74µs ? ?/sec 1.00 714.6±31.68µs ? ?/sec
variant_get_primitive 1.00 563.1±6.49ns ? ?/sec 1.00 562.6±7.17ns ? ?/sec
variant_get_shredded_utf8 1.00 715.6±5.50ns ? ?/sec 1.00 716.2±6.45ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.07 37.6±0.52ms ? ?/sec 1.00 35.1±0.87ms ? ?/sec
variant_get_utf8_from_date 1.29 569.7±4.93µs ? ?/sec 1.00 442.5±12.76µs ? ?/sec
variant_get_utf8_from_decimal 1.25 473.1±6.15µs ? ?/sec 1.00 378.2±27.79µs ? ?/sec
variant_get_utf8_from_float 1.32 447.5±1.39µs ? ?/sec 1.00 339.2±24.65µs ? ?/sec
variant_get_utf8_from_int 1.32 447.6±1.45µs ? ?/sec 1.00 340.2±24.75µs ? ?/sec
variant_get_utf8_from_list 1.00 659.2±12.62µs ? ?/sec 1.06 699.0±29.32µs ? ?/sec
variant_get_utf8_from_map_in_list 1.00 680.8±12.54µs ? ?/sec 1.07 725.2±29.89µs ? ?/sec
variant_get_utf8_from_time 1.18 776.1±2.61µs ? ?/sec 1.00 658.8±11.44µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.30 571.3±3.08µs ? ?/sec 1.00 440.7±16.97µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.30 571.3±3.11µs ? ?/sec 1.00 440.4±16.74µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.26 493.8±1.27µs ? ?/sec 1.00 392.6±25.88µs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_kernels — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-ufagysz2-36d0891d-j7hk
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6111106677-3301-b6zvd 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 395.1s |
| Peak memory | 408.8 MiB |
| Avg memory | 271.3 MiB |
| CPU user | 388.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 405.1s |
| Peak memory | 348.8 MiB |
| Avg memory | 272.4 MiB |
| CPU user | 399.3s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6111106677:finish:911238b3faba30291d755564fb7f487f -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635038
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
resources:
arch: amd64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110734830
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6110635211)
**Target:** variant_validation
Comparing variant-typestate-views (340f702b52ef399659887c4e79edf9541098264a) to 4c67abc (merge-base) [diff](https://github.com/apache/arrow-rs/compare/4c67abc4505344425fd9d5172037bd0f468f8b42..340f702b52ef399659887c4e79edf9541098264a)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
bench_validate_complex_object 1.00 182.1±2.90µs ? ?/sec 1.00 182.8±2.39µs ? ?/sec
bench_validate_large_nested_list 1.00 20.2±0.52ms ? ?/sec 1.00 20.2±0.43ms ? ?/sec
bench_validate_large_object 1.00 55.2±2.18ms ? ?/sec 1.00 55.3±1.98ms ? ?/sec
metadata_iter/128 1.02 994.4±70.98ns 122.8 MElem/sec 1.00 976.6±15.69ns 125.0 MElem/sec
metadata_iter/32 1.02 253.5±17.33ns 120.4 MElem/sec 1.00 248.2±3.60ns 122.9 MElem/sec
metadata_iter/8 1.00 60.9±3.12ns 125.4 MElem/sec 1.00 61.1±1.08ns 124.8 MElem/sec
object_field_name/128 1.00 1178.3±32.95ns 103.6 MElem/sec 1.26 1485.1±6.52ns 82.2 MElem/sec
object_field_name/32 1.00 299.6±7.53ns 101.9 MElem/sec 1.25 374.5±0.87ns 81.5 MElem/sec
object_field_name/8 1.00 74.0±1.84ns 103.1 MElem/sec 1.27 94.1±0.57ns 81.1 MElem/sec
object_get_by_name/128 1.16 110.3±0.90ns ? ?/sec 1.00 94.7±0.41ns ? ?/sec
object_get_by_name/32 1.08 72.7±1.03ns ? ?/sec 1.00 67.6±0.25ns ? ?/sec
object_get_by_name/8 1.04 59.8±0.92ns ? ?/sec 1.00 57.7±0.28ns ? ?/sec
object_iter/128 1.00 2.6±0.05µs 46.7 MElem/sec 1.02 2.7±0.03µs 46.0 MElem/sec
object_iter/32 1.00 658.8±11.72ns 46.3 MElem/sec 1.01 667.4±8.16ns 45.7 MElem/sec
object_iter/8 1.00 168.9±2.26ns 45.2 MElem/sec 1.00 169.2±2.06ns 45.1 MElem/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>variant_validation — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-1fiesfzt-fd3af563-2trb
**Instance:** c4a-highcpu-48 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6110635211-3298-fnfh9 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 3 MiB (48 instances)
L1i cache: 3 MiB (48 instances)
L2 cache: 96 MiB (48 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.1 MiB |
| CPU user | 151.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 145.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.7 MiB |
| CPU user | 141.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6110635211:finish:abe1c8d16dd9fba3df714aa45d82b87a -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6111083003
run benchmark variant_validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(variant): read only what each access uses via header-only views [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11511:
URL: https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256786953
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11511#issuecomment-6256786133)
**Target:** variant_kernels
**Sharding factor: 1** (1 workers).
Comparing variant-typestate-views (c5c21219423b575cfa76bbea6bf7c3ade5290de5) to 33d338d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/33d338ddf661b6e85364a9b0bd4d367c3468257d..c5c21219423b575cfa76bbea6bf7c3ade5290de5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: ^(variant_get|shred_variant)
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6256786133:start:33d34ed35ea45ebc4f1969e54faaafa7 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org