You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "friendlymatthew (via GitHub)" <gi...@apache.org> on 2026/07/16 21:20:03 UTC
[PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
friendlymatthew opened a new pull request, #10358:
URL: https://github.com/apache/arrow-rs/pull/10358
# Rationale for this change
Based on benchmark https://github.com/apache/arrow-rs/pull/10357
<img width="1512" height="382" alt="Screenshot 2026-07-16 at 5 15 20 PM" src="https://github.com/user-attachments/assets/74425745-85ea-4411-88f7-6845f6dd8ad7" />
This PR speeds up field extraction from unshredded variant objects by avoiding repeated field name decoding for every row
Variant object keys are stored as numeric field ids that map to a metadata dictionary. `variant_get` currently stores a named path by repeatedly mapping those ids back to strings and comparing the strings for each row, _even when a batch shares one metadata dictionary_. This repeated per-row work dominates when dealing with wide dictionaries
This optimization adds metadata scoped field id lookups, caches parsed metadata, and resolves named path elements to ids once per metadata dictionary before scanning rows. It re-resolves paths when the metadata changes and retains name lookup for unsorted dictionaries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733453786
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733354689)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733354689-2443-nsp7h 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733548715
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356216)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733356216-2447-xfzpf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732873463
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732803010)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732803010-2432-ncwh8 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5529381147
@alamb
- This PR optimizes read path
- #10882 optimizes Variant transform paths like in `shred_variant`, so same idea but complement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732801519
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732766211
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733571417
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733359404)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733359404-2451-gvn5v 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733353785
run benchmark variant_validation variant_builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5443231081
FYI @adriangb -- is this one related to your recent work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732883185
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732802232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.00 21.0±0.11ms ? ?/sec 1.00 20.9±0.10ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.00 26.2±0.18ms ? ?/sec 1.00 26.3±0.18ms ? ?/sec
variant_get_primitive 1.00 563.7±3.25ns ? ?/sec 1.00 566.1±5.60ns ? ?/sec
variant_get_shredded_utf8 1.00 712.7±3.70ns ? ?/sec 1.00 710.3±6.06ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.00 45.7±0.45ms ? ?/sec 1.00 45.6±0.38ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 305.1 MiB |
| Avg memory | 18.6 MiB |
| CPU user | 57.6s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 318.3 MiB |
| Avg memory | 18.8 MiB |
| CPU user | 57.0s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800756
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360105
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733415172
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733324159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.01 182.6±2.38µs ? ?/sec 1.00 181.6±2.56µs ? ?/sec
bench_validate_large_nested_list 1.01 20.2±0.44ms ? ?/sec 1.00 20.1±0.50ms ? ?/sec
bench_validate_large_object 1.01 55.5±2.17ms ? ?/sec 1.00 55.3±2.18ms ? ?/sec
metadata_iter/128 1.00 1000.2±63.71ns 122.0 MElem/sec 1.00 996.5±102.94ns 122.5 MElem/sec
metadata_iter/32 1.01 256.0±17.78ns 119.2 MElem/sec 1.00 253.6±26.08ns 120.3 MElem/sec
metadata_iter/8 1.00 61.2±3.10ns 124.7 MElem/sec 1.00 61.2±6.38ns 124.7 MElem/sec
object_field_name/128 1.07 1266.9±204.97ns 96.4 MElem/sec 1.00 1178.6±33.20ns 103.6 MElem/sec
object_field_name/32 1.07 321.5±50.27ns 94.9 MElem/sec 1.00 299.8±6.79ns 101.8 MElem/sec
object_field_name/8 1.08 79.6±13.67ns 95.8 MElem/sec 1.00 73.9±2.02ns 103.3 MElem/sec
object_get_by_name/128 1.01 111.3±0.70ns ? ?/sec 1.00 110.3±0.91ns ? ?/sec
object_get_by_name/32 1.01 73.9±0.73ns ? ?/sec 1.00 72.9±1.07ns ? ?/sec
object_get_by_name/8 1.02 60.8±0.79ns ? ?/sec 1.00 59.7±0.88ns ? ?/sec
object_iter/128 1.03 2.7±0.09µs 45.3 MElem/sec 1.00 2.6±0.05µs 46.5 MElem/sec
object_iter/32 1.03 678.4±23.79ns 45.0 MElem/sec 1.00 659.6±12.13ns 46.3 MElem/sec
object_iter/8 1.03 174.5±5.52ns 43.7 MElem/sec 1.00 168.9±2.31ns 45.2 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 150.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.7 MiB |
| CPU user | 152.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733551430
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733355457)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.00 181.9±2.35µs ? ?/sec 1.00 181.6±2.55µs ? ?/sec
bench_validate_large_nested_list 1.00 20.2±0.44ms ? ?/sec 1.00 20.1±0.50ms ? ?/sec
bench_validate_large_object 1.01 55.7±2.21ms ? ?/sec 1.00 55.3±2.20ms ? ?/sec
metadata_iter/128 1.01 993.9±55.75ns 122.8 MElem/sec 1.00 986.8±57.96ns 123.7 MElem/sec
metadata_iter/32 1.01 254.0±14.06ns 120.2 MElem/sec 1.00 252.0±15.72ns 121.1 MElem/sec
metadata_iter/8 1.01 61.2±3.21ns 124.6 MElem/sec 1.00 60.9±3.18ns 125.3 MElem/sec
object_field_name/128 1.00 1183.3±36.11ns 103.2 MElem/sec 1.00 1178.9±32.69ns 103.5 MElem/sec
object_field_name/32 1.00 300.8±7.40ns 101.4 MElem/sec 1.00 299.9±7.10ns 101.8 MElem/sec
object_field_name/8 1.00 74.3±2.06ns 102.7 MElem/sec 1.00 73.9±1.97ns 103.2 MElem/sec
object_get_by_name/128 1.00 109.7±0.91ns ? ?/sec 1.00 109.8±0.88ns ? ?/sec
object_get_by_name/32 1.00 72.8±0.97ns ? ?/sec 1.01 73.3±1.13ns ? ?/sec
object_get_by_name/8 1.00 59.6±0.90ns ? ?/sec 1.00 59.7±0.93ns ? ?/sec
object_iter/128 1.00 2.6±0.04µs 46.6 MElem/sec 1.00 2.6±0.05µs 46.6 MElem/sec
object_iter/32 1.00 659.6±9.42ns 46.3 MElem/sec 1.00 659.2±11.61ns 46.3 MElem/sec
object_iter/8 1.00 169.2±2.23ns 45.1 MElem/sec 1.00 169.1±2.45ns 45.1 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 150.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.6 MiB |
| CPU user | 150.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733541054
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733355457)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.00 21.2±0.08ms ? ?/sec 1.00 21.2±0.08ms ? ?/sec
bench_object_field_names_reverse_order 1.00 12.9±0.87ms ? ?/sec 1.08 13.9±0.43ms ? ?/sec
bench_object_list_partially_same_schema 1.00 1062.5±12.75µs ? ?/sec 1.00 1061.9±14.12µs ? ?/sec
bench_object_list_same_schema 1.00 21.4±0.16ms ? ?/sec 1.00 21.4±0.11ms ? ?/sec
bench_object_list_unknown_schema 1.00 10.7±0.03ms ? ?/sec 1.00 10.8±0.04ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.04 33.5±0.06ms ? ?/sec 1.00 32.2±0.10ms ? ?/sec
bench_object_unknown_schema 1.00 13.2±0.05ms ? ?/sec 1.00 13.2±0.03ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.01 40.1±0.30µs ? ?/sec 1.00 39.9±0.41µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.07µs ? ?/sec 1.01 4.3±0.06µs ? ?/sec
validation/validated_construction 1.01 68.2±1.32µs ? ?/sec 1.00 67.4±1.00µs ? ?/sec
validation/validation_cost 1.00 57.5±1.13µs ? ?/sec 1.00 57.3±0.80µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 17.0 MiB |
| CPU user | 133.0s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.8 MiB |
| CPU user | 132.7s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5109372972
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733536697
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356216)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.00 181.6±2.57µs ? ?/sec 1.00 181.6±2.54µs ? ?/sec
bench_validate_large_nested_list 1.00 20.1±0.50ms ? ?/sec 1.00 20.1±0.58ms ? ?/sec
bench_validate_large_object 1.00 55.3±2.18ms ? ?/sec 1.00 55.5±2.40ms ? ?/sec
metadata_iter/128 1.00 987.4±61.87ns 123.6 MElem/sec 1.01 999.4±68.42ns 122.1 MElem/sec
metadata_iter/32 1.00 252.1±15.36ns 121.1 MElem/sec 1.01 253.7±17.33ns 120.3 MElem/sec
metadata_iter/8 1.00 61.0±3.37ns 125.1 MElem/sec 1.00 61.0±3.59ns 125.0 MElem/sec
object_field_name/128 1.00 1178.2±33.24ns 103.6 MElem/sec 1.00 1178.7±39.10ns 103.6 MElem/sec
object_field_name/32 1.00 299.7±6.97ns 101.8 MElem/sec 1.00 299.7±8.27ns 101.8 MElem/sec
object_field_name/8 1.00 73.8±1.83ns 103.3 MElem/sec 1.00 74.0±2.21ns 103.1 MElem/sec
object_get_by_name/128 1.00 110.5±1.07ns ? ?/sec 1.00 110.0±0.89ns ? ?/sec
object_get_by_name/32 1.00 73.0±1.10ns ? ?/sec 1.00 72.7±1.01ns ? ?/sec
object_get_by_name/8 1.00 59.8±0.94ns ? ?/sec 1.01 60.2±0.94ns ? ?/sec
object_iter/128 1.00 2.6±0.05µs 46.6 MElem/sec 1.00 2.6±0.04µs 46.3 MElem/sec
object_iter/32 1.00 658.2±11.56ns 46.4 MElem/sec 1.01 662.6±11.32ns 46.1 MElem/sec
object_iter/8 1.00 169.0±2.37ns 45.1 MElem/sec 1.00 169.7±2.16ns 45.0 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 12.9 MiB |
| CPU user | 148.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.6 MiB |
| CPU user | 150.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733527902
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733354689)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.00 182.4±2.58µs ? ?/sec 1.00 181.7±2.56µs ? ?/sec
bench_validate_large_nested_list 1.00 20.1±0.50ms ? ?/sec 1.00 20.1±0.51ms ? ?/sec
bench_validate_large_object 1.00 55.3±2.18ms ? ?/sec 1.00 55.3±2.18ms ? ?/sec
metadata_iter/128 1.01 992.5±63.13ns 123.0 MElem/sec 1.00 987.3±57.77ns 123.6 MElem/sec
metadata_iter/32 1.00 252.2±15.90ns 121.0 MElem/sec 1.00 251.4±15.40ns 121.4 MElem/sec
metadata_iter/8 1.00 60.9±3.28ns 125.4 MElem/sec 1.00 61.0±3.51ns 125.1 MElem/sec
object_field_name/128 1.00 1177.5±34.56ns 103.7 MElem/sec 1.00 1173.4±39.31ns 104.0 MElem/sec
object_field_name/32 1.00 299.4±7.43ns 101.9 MElem/sec 1.00 298.8±8.03ns 102.1 MElem/sec
object_field_name/8 1.00 73.8±1.89ns 103.4 MElem/sec 1.00 73.8±2.05ns 103.3 MElem/sec
object_get_by_name/128 1.03 112.8±6.94ns ? ?/sec 1.00 109.8±0.91ns ? ?/sec
object_get_by_name/32 1.03 74.6±3.67ns ? ?/sec 1.00 72.6±1.04ns ? ?/sec
object_get_by_name/8 1.02 61.0±2.94ns ? ?/sec 1.00 59.7±0.88ns ? ?/sec
object_iter/128 1.00 2.6±0.05µs 46.4 MElem/sec 1.00 2.6±0.05µs 46.6 MElem/sec
object_iter/32 1.00 657.5±10.98ns 46.4 MElem/sec 1.00 659.0±11.77ns 46.3 MElem/sec
object_iter/8 1.00 168.9±2.38ns 45.2 MElem/sec 1.00 168.8±2.31ns 45.2 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 151.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.6 MiB |
| CPU user | 150.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733355457
run benchmark variant_validation variant_builder
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733324159
run benchmark variant_validation variant_builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733444471
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733353785)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.01 21.4±0.09ms ? ?/sec 1.00 21.2±0.10ms ? ?/sec
bench_object_field_names_reverse_order 1.00 13.2±0.90ms ? ?/sec 1.05 13.9±0.37ms ? ?/sec
bench_object_list_partially_same_schema 1.01 1067.2±15.53µs ? ?/sec 1.00 1058.1±9.13µs ? ?/sec
bench_object_list_same_schema 1.00 21.4±0.12ms ? ?/sec 1.00 21.4±0.11ms ? ?/sec
bench_object_list_unknown_schema 1.00 10.8±0.04ms ? ?/sec 1.00 10.8±0.04ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.05 33.5±0.08ms ? ?/sec 1.00 31.9±0.09ms ? ?/sec
bench_object_unknown_schema 1.00 13.2±0.04ms ? ?/sec 1.00 13.1±0.05ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.01 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.00 39.8±0.31µs ? ?/sec 1.00 40.0±0.34µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.07µs ? ?/sec 1.01 4.3±0.07µs ? ?/sec
validation/validated_construction 1.00 67.3±1.03µs ? ?/sec 1.00 67.4±0.92µs ? ?/sec
validation/validation_cost 1.00 57.2±0.81µs ? ?/sec 1.00 57.4±0.91µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.9 MiB |
| CPU user | 132.0s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.7 MiB |
| CPU user | 132.7s |
| CPU sys | 1.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733462080
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356216)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733356216-2446-8qkhj 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732802232
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732871447
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732801519)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.00 21.0±0.10ms ? ?/sec 1.00 20.9±0.09ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.00 26.2±0.15ms ? ?/sec 1.00 26.3±0.16ms ? ?/sec
variant_get_primitive 1.00 560.7±5.04ns ? ?/sec 1.01 564.5±4.11ns ? ?/sec
variant_get_shredded_utf8 1.00 708.4±5.18ns ? ?/sec 1.00 707.7±8.56ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.01 46.1±0.55ms ? ?/sec 1.00 45.6±0.47ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 357.3 MiB |
| Avg memory | 18.9 MiB |
| CPU user | 54.7s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 252.8 MiB |
| Avg memory | 17.7 MiB |
| CPU user | 57.0s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733617576
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356216)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.02 21.4±0.61ms ? ?/sec 1.00 21.0±0.23ms ? ?/sec
bench_object_field_names_reverse_order 1.00 12.9±1.05ms ? ?/sec 1.05 13.6±0.42ms ? ?/sec
bench_object_list_partially_same_schema 1.00 1058.4±11.42µs ? ?/sec 1.00 1057.4±9.72µs ? ?/sec
bench_object_list_same_schema 1.00 21.3±0.16ms ? ?/sec 1.02 21.7±0.51ms ? ?/sec
bench_object_list_unknown_schema 1.00 10.8±0.05ms ? ?/sec 1.00 10.8±0.05ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.03ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.05 33.7±0.07ms ? ?/sec 1.00 32.0±0.08ms ? ?/sec
bench_object_unknown_schema 1.00 13.1±0.04ms ? ?/sec 1.00 13.2±0.05ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.00 40.0±0.34µs ? ?/sec 1.00 40.0±0.34µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.08µs ? ?/sec 1.00 4.4±0.08µs ? ?/sec
validation/validated_construction 1.00 68.1±1.53µs ? ?/sec 1.00 68.3±1.28µs ? ?/sec
validation/validation_cost 1.00 57.0±0.90µs ? ?/sec 1.01 57.5±1.05µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.8 MiB |
| CPU user | 132.1s |
| CPU sys | 1.8s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.8 MiB |
| CPU user | 132.8s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360851
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733352952
run benchmark variant_validation variant_builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733547870
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733356982-2448-lb4wf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732803010
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733703583
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360105)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.02 19.5±0.16ms ? ?/sec 1.00 19.2±0.14ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.01 147.2±0.52ms ? ?/sec 1.00 146.4±0.26ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.00 306.7±0.94ms ? ?/sec 1.00 306.7±1.19ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 218.7±0.69ms ? ?/sec 1.00 219.2±1.28ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.00 6.4±0.04ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
variant_array_builder_build_262k_small_values 1.00 2.7±0.17ms ? ?/sec 1.01 2.7±0.22ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 224.5 MiB |
| CPU user | 132.3s |
| CPU sys | 5.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 226.6 MiB |
| CPU user | 131.6s |
| CPU sys | 4.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732866001
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800033)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.02 21.3±0.14ms ? ?/sec 1.00 20.9±0.09ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.02 26.8±0.14ms ? ?/sec 1.00 26.3±0.18ms ? ?/sec
variant_get_primitive 1.00 565.3±4.25ns ? ?/sec 1.00 563.0±6.37ns ? ?/sec
variant_get_shredded_utf8 1.02 719.1±10.13ns ? ?/sec 1.00 707.9±3.82ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.00 16.0±0.08ms ? ?/sec 2.89 46.1±0.65ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 302.7 MiB |
| Avg memory | 18.5 MiB |
| CPU user | 57.6s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 364.4 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 52.9s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732828954
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732766211)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.01 21.2±0.21ms ? ?/sec 1.00 20.9±0.11ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.02 26.7±0.29ms ? ?/sec 1.00 26.3±0.18ms ? ?/sec
variant_get_primitive 1.00 566.8±5.95ns ? ?/sec 1.00 568.1±7.57ns ? ?/sec
variant_get_shredded_utf8 1.00 706.2±4.09ns ? ?/sec 1.00 706.7±4.47ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.00 16.0±0.08ms ? ?/sec 2.87 45.8±0.37ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 396.5 MiB |
| Avg memory | 20.1 MiB |
| CPU user | 57.7s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 76.7 MiB |
| Avg memory | 14.5 MiB |
| CPU user | 51.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732835665
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800756)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732800756-2429-bvxhh 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733380651
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733353785)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733353785-2441-4qns7 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732835262
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800033)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732800033-2428-kgtkd 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732839806
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732801519)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732801519-2430-9sg9k 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733703724
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360851)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.02 19.5±0.14ms ? ?/sec 1.00 19.2±0.14ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.00 147.1±0.23ms ? ?/sec 1.00 146.7±0.25ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.00 306.5±0.60ms ? ?/sec 1.00 306.6±0.62ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 222.3±4.01ms ? ?/sec 1.03 229.8±1.47ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.00 6.4±0.07ms ? ?/sec 1.00 6.4±0.05ms ? ?/sec
variant_array_builder_build_262k_small_values 1.01 2.5±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 223.0 MiB |
| CPU user | 130.8s |
| CPU sys | 5.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 224.0 MiB |
| CPU user | 129.9s |
| CPU sys | 4.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800033
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733379676
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733353785)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733353785-2440-s9z5x 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733357732
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733472656
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733355457)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733355457-2445-5pqh4 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733640261
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360105)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733360105-2452-nhq5j 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733436629
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733352952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.00 181.4±2.53µs ? ?/sec 1.00 181.8±2.36µs ? ?/sec
bench_validate_large_nested_list 1.00 20.1±0.49ms ? ?/sec 1.01 20.3±0.50ms ? ?/sec
bench_validate_large_object 1.00 55.2±2.43ms ? ?/sec 1.01 55.7±2.19ms ? ?/sec
metadata_iter/128 1.00 996.9±69.14ns 122.5 MElem/sec 1.00 1000.4±62.85ns 122.0 MElem/sec
metadata_iter/32 1.00 255.4±18.49ns 119.5 MElem/sec 1.00 254.2±15.73ns 120.0 MElem/sec
metadata_iter/8 1.00 60.8±3.51ns 125.6 MElem/sec 1.01 61.3±3.60ns 124.5 MElem/sec
object_field_name/128 1.00 1185.2±33.71ns 103.0 MElem/sec 1.00 1183.6±36.10ns 103.1 MElem/sec
object_field_name/32 1.00 300.7±6.80ns 101.5 MElem/sec 1.00 301.0±7.52ns 101.4 MElem/sec
object_field_name/8 1.00 74.2±2.05ns 102.8 MElem/sec 1.00 74.2±2.17ns 102.8 MElem/sec
object_get_by_name/128 1.02 111.9±0.65ns ? ?/sec 1.00 109.8±0.95ns ? ?/sec
object_get_by_name/32 1.02 73.8±0.81ns ? ?/sec 1.00 72.5±0.90ns ? ?/sec
object_get_by_name/8 1.01 60.4±0.72ns ? ?/sec 1.00 59.7±0.94ns ? ?/sec
object_iter/128 1.00 2.6±0.03µs 46.8 MElem/sec 1.00 2.6±0.05µs 46.5 MElem/sec
object_iter/32 1.00 658.1±8.20ns 46.4 MElem/sec 1.00 659.0±10.93ns 46.3 MElem/sec
object_iter/8 1.00 169.0±1.55ns 45.1 MElem/sec 1.00 169.1±2.17ns 45.1 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 148.8s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 150.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.9 MiB |
| CPU user | 148.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733405245
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733352952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733352952-2439-92g65 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733636359
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733357732)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.05 20.1±0.28ms ? ?/sec 1.00 19.1±0.16ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.00 146.9±0.29ms ? ?/sec 1.00 147.5±0.53ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.00 309.3±2.55ms ? ?/sec 1.00 308.6±1.25ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 220.3±0.89ms ? ?/sec 1.00 219.8±1.49ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.00 6.4±0.05ms ? ?/sec 1.00 6.4±0.11ms ? ?/sec
variant_array_builder_build_262k_small_values 1.00 2.7±0.15ms ? ?/sec 1.00 2.7±0.14ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 223.8 MiB |
| CPU user | 131.1s |
| CPU sys | 5.8s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 226.9 MiB |
| CPU user | 131.4s |
| CPU sys | 4.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732852210
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732802232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732802232-2431-cv8df 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733359404
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733354689
run benchmark variant_validation variant_builder
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356982
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733358572
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733724577
## Benchmark summary
`variant_get` on an unshredded object path is **2.89x faster**: 16.0ms against 45.8ms for 262k rows.
Across 39 benchmarks in three suites: one gets much faster, three get about 2% slower, and 35 are flat.
| bench | change runs | control runs | verdict |
| --- | --- | --- | --- |
| `variant_get_unshredded_object_path_262k_rows` | 0.35 / 0.35 / 0.34 | 1.01 / 1.00 / 1.00 | **2.89x faster** |
| `shred_variant_unmatched_object_8k_rows` | 1.02 / 1.02 / 1.02 | 1.00 / 1.00 / 1.00 | 2% slower |
| `shred_variant_partial_object_8k_rows` | 1.01 / 1.02 / 1.02 | 1.00 / 1.00 / 1.00 | 2% slower |
| `batch_json_string_to_variant json_list 8k string` | 1.04 / 1.04 / 1.05 | 1.02 / 1.02 / 1.02 | 2% slower |
| the other 35 | - | - | flat |
Each number is the ratio of this branch to its merge base. Every job has a paired main-vs-main control job, because several of these benchmarks move by 1 to 5% on identical code. Twelve change jobs and twelve control jobs in total.
The PR title says 79%. That number is from the original measurement in July. The correct number for this branch on this runner is 2.89x, which is a 65% decrease in latency.
### About the three benchmarks that get slower
None of the three can reach the code this PR changes, and the direct microbenchmarks of the changed functions are flat:
- `object_field_name` is a tight loop on `VariantObject::field_name`, the function this PR splits into `try_field_name` plus `try_field_id`. It is flat: 1.00 and 1.01 against a control at 1.00.
- `object_get_by_name` and `object_iter` are also flat.
- `json_to_variant` and `shred_variant` write variants. They do not call the new path resolution at all.
All three benchmarks that move are in `parquet-variant-compute`, which this PR grows by about 180 lines. All 22 benchmarks in `parquet-variant`, which grows by about 30 lines of non-test code, are flat. That points at code placement rather than a cost in the new logic.
Two things I ruled out:
- **Enum growth.** `VariantPathRowBuilder` grows from 32 to 88 bytes, and it is one variant of the `VariantToArrowRowBuilder` enum that `shred_variant` uses. That enum is 224 bytes both on main and here, because a different variant is the largest one.
- **An `#[inline]` on `try_field_id`.** It changes the emitted IR, but the benchmark that measures the split is already flat, so there was nothing to recover. I dropped it.
These benchmarks are very sensitive to placement. On my laptop, `variant_get_shredded_utf8` reported +7.8% for this branch, and then +14.0% after I moved the new functions to a different place in the same file and changed nothing else. On this runner it is flat.
My suggestion is to take about 2% on three benchmarks this PR does not target, for 2.89x on the one it does. I am happy to look further if you would rather not.
<details><summary>The 35 flat benchmarks, and how these runs were made</summary>
- `variant_kernels`: `variant_get_primitive`, `variant_get_shredded_utf8`, `variant_array_builder_build_262k_small_values`, and four more `batch_json_string_to_variant` cases (`repeated_struct 8k string`, `random_json(2633 bytes per document)`, `object - 1 depth(100 fields)`, `object - 1 depth(200 fields)`)
- `variant_validation`: `bench_validate_large_object`, `bench_validate_complex_object`, `bench_validate_large_nested_list`, `metadata_iter/{8,32,128}`, `object_field_name/{8,32,128}`, `object_iter/{8,32,128}`, `object_get_by_name/{8,32,128}`
- `variant_builder`: `bench_object_same_schema`, `bench_object_field_names_reverse_order`, `bench_object_unknown_schema`, `bench_object_partially_same_schema`, `bench_object_list_same_schema`, `bench_object_list_unknown_schema`, `bench_object_list_partially_same_schema`, `bench_extend_metadata_builder`, `validation/{validated_construction,unvalidated_construction,validation_cost}`, `iteration/{validated_iteration,unvalidated_fallible_iteration}`
Two of those need a note, because the ratio on its own is misleading:
- `bench_object_same_schema` reads a steady 1.05 in all three change runs, with error bars of 0.2%. Its three control runs read 1.03, 1.04 and 1.05 on identical code. The runner benchmarks the baseline side first on the same pod, and this benchmark is 3 to 5% slower on the second side whatever is in it. It is flat.
- One `variant_validation` job reported `object_field_name` at 1.07 to 1.08, but with 17% error bars on one side only (79.6+/-13.67ns against 73.9+/-2.02ns). Two other runs of the same commit are flat with 3% error bars. That pod was disturbed, so I discarded the run.
Run configuration, on the `adriangbot` GKE runner (`c4a-highmem-16`, 12 vCPU, Neoverse-V2), three change jobs and three control jobs of each:
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
```yaml
run benchmark variant_validation variant_builder
```
A control job pins both sides to the merge base, `1a6630bc2c198955a9d24b34ca560dcbb9167a7f`:
```yaml
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
```
A control job compares two identical binaries. It gives the noise floor and it shows the tilt above, but it cannot separate a code placement effect from a real cost, because placement only differs between different binaries.
`parquet-variant` and `parquet-variant-compute` hold every variant benchmark in the repository. The `parquet` crate has none, so there is nothing else to measure.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5878659222
@adriangb @friendlymatthew can you merge main and rerun the bench?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732866321
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732800756)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.02 21.3±0.12ms ? ?/sec 1.00 20.9±0.12ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.02 26.9±0.14ms ? ?/sec 1.00 26.3±0.20ms ? ?/sec
variant_get_primitive 1.01 566.4±4.16ns ? ?/sec 1.00 562.3±3.12ns ? ?/sec
variant_get_shredded_utf8 1.00 707.5±4.18ns ? ?/sec 1.01 716.9±6.11ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.00 16.1±0.05ms ? ?/sec 2.90 46.6±0.76ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 270.7 MiB |
| Avg memory | 17.5 MiB |
| CPU user | 54.7s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 363.4 MiB |
| Avg memory | 20.2 MiB |
| CPU user | 53.0s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732902943
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732803010)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
shred_variant_partial_object_8k_rows 1.00 20.9±0.09ms ? ?/sec 1.00 20.9±0.09ms ? ?/sec
shred_variant_unmatched_object_8k_rows 1.00 26.2±0.19ms ? ?/sec 1.00 26.2±0.20ms ? ?/sec
variant_get_primitive 1.01 569.4±7.82ns ? ?/sec 1.00 562.6±3.84ns ? ?/sec
variant_get_shredded_utf8 1.01 714.7±8.53ns ? ?/sec 1.00 710.4±6.16ns ? ?/sec
variant_get_unshredded_object_path_262k_rows 1.00 45.6±0.38ms ? ?/sec 1.00 45.6±0.42ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 405.7 MiB |
| Avg memory | 20.3 MiB |
| CPU user | 57.8s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 60.0s |
| Peak memory | 287.7 MiB |
| Avg memory | 18.3 MiB |
| CPU user | 57.0s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733368963
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733352952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733352952-2438-kcvxg 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356216
run benchmark variant_validation variant_builder
baseline:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
changed:
ref: 1a6630bc2c198955a9d24b34ca560dcbb9167a7f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733426095
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733324159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.02 21.3±0.07ms ? ?/sec 1.00 21.0±0.07ms ? ?/sec
bench_object_field_names_reverse_order 1.00 12.9±0.85ms ? ?/sec 1.03 13.3±0.30ms ? ?/sec
bench_object_list_partially_same_schema 1.00 1065.9±13.06µs ? ?/sec 1.00 1063.3±11.52µs ? ?/sec
bench_object_list_same_schema 1.00 21.3±0.11ms ? ?/sec 1.00 21.4±0.10ms ? ?/sec
bench_object_list_unknown_schema 1.00 10.8±0.03ms ? ?/sec 1.00 10.8±0.04ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.05 33.7±0.08ms ? ?/sec 1.00 32.0±0.08ms ? ?/sec
bench_object_unknown_schema 1.00 13.2±0.04ms ? ?/sec 1.00 13.2±0.04ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.01 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.00 40.1±0.19µs ? ?/sec 1.00 40.0±0.34µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.07µs ? ?/sec 1.01 4.3±0.08µs ? ?/sec
validation/validated_construction 1.00 67.3±0.93µs ? ?/sec 1.00 67.5±1.09µs ? ?/sec
validation/validation_cost 1.00 57.0±0.92µs ? ?/sec 1.02 58.4±1.52µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 17.0 MiB |
| CPU user | 132.3s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.7 MiB |
| CPU user | 132.7s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733619852
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733356982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.04 19.9±0.23ms ? ?/sec 1.00 19.1±0.16ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.00 147.0±0.24ms ? ?/sec 1.00 146.5±0.37ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.00 307.4±0.67ms ? ?/sec 1.00 307.1±0.73ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 223.0±3.79ms ? ?/sec 1.03 230.8±1.41ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.01 6.4±0.05ms ? ?/sec 1.00 6.4±0.05ms ? ?/sec
variant_array_builder_build_262k_small_values 1.03 2.7±0.02ms ? ?/sec 1.00 2.6±0.04ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 225.3 MiB |
| CPU user | 131.3s |
| CPU sys | 5.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 227.6 MiB |
| CPU user | 131.6s |
| CPU sys | 4.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733555871
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733358572)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733358572-2450-97qzv 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#discussion_r3669237405
##########
parquet-variant-compute/benches/variant_kernels.rs:
##########
@@ -223,6 +239,40 @@ fn create_shredded_utf8_variant_array(size: usize) -> VariantArray {
.expect("created struct should be a valid shredded variant")
}
+fn create_unshredded_object_variant_array(size: usize) -> VariantArray {
Review Comment:
It would be helpful if you split the change to the benchmark off into a separate PR I can merge it and then use the benchmarking scripts
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -241,27 +242,72 @@ fn shredded_get_path(
} else {
as_field.map(|f| f.data_type())
};
+ let has_path = !path.is_empty();
let mut builder = make_variant_to_arrow_row_builder(
target.metadata_column(),
path,
as_type,
cast_options,
target.len(),
)?;
- for i in 0..target.len() {
- if target.is_null(i) {
- builder.append_null()?;
- } else if !cast_options.safe {
- let value = target.try_value(i)?;
- builder.append_value(value)?;
- } else {
- let _ = match target.try_value(i) {
- Ok(v) => builder.append_value(v)?,
- Err(_) => {
- builder.append_null()?;
- false // add this to make match arms have the same return type
+
+ if has_path
+ && target.typed_value_column().is_none()
+ && let Some(value_column) = target.value_column()
+ {
+ let metadata_column = target.metadata_column();
Review Comment:
Given that this function is already pretty long, could you please try and refactor some of the logic into smaller documented functions that each do some particular operation?
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -241,27 +242,72 @@ fn shredded_get_path(
} else {
as_field.map(|f| f.data_type())
};
+ let has_path = !path.is_empty();
let mut builder = make_variant_to_arrow_row_builder(
target.metadata_column(),
path,
as_type,
cast_options,
target.len(),
)?;
- for i in 0..target.len() {
- if target.is_null(i) {
- builder.append_null()?;
- } else if !cast_options.safe {
- let value = target.try_value(i)?;
- builder.append_value(value)?;
- } else {
- let _ = match target.try_value(i) {
- Ok(v) => builder.append_value(v)?,
- Err(_) => {
- builder.append_null()?;
- false // add this to make match arms have the same return type
+
+ if has_path
Review Comment:
Could you add some comments to explain the rationale for this condition / fast path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732797802
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5732766211)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5732766211-2427-m9qcj 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_get|shred_variant)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733365070
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733324159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733324159-2437-7khc6 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_builder
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733349649
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733324159)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733324159-2436-rzpcz 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733449896
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733353785)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_validate_complex_object 1.00 181.5±2.39µs ? ?/sec 1.00 181.8±2.48µs ? ?/sec
bench_validate_large_nested_list 1.00 20.1±0.43ms ? ?/sec 1.00 20.1±0.50ms ? ?/sec
bench_validate_large_object 1.00 55.5±2.24ms ? ?/sec 1.00 55.3±2.18ms ? ?/sec
metadata_iter/128 1.01 1002.6±64.22ns 121.8 MElem/sec 1.00 989.9±61.26ns 123.3 MElem/sec
metadata_iter/32 1.01 255.5±16.93ns 119.4 MElem/sec 1.00 251.8±15.75ns 121.2 MElem/sec
metadata_iter/8 1.00 60.7±2.84ns 125.7 MElem/sec 1.00 60.6±3.26ns 125.8 MElem/sec
object_field_name/128 1.01 1187.9±33.21ns 102.8 MElem/sec 1.00 1178.7±33.40ns 103.6 MElem/sec
object_field_name/32 1.00 301.3±6.86ns 101.3 MElem/sec 1.00 300.8±7.31ns 101.5 MElem/sec
object_field_name/8 1.01 74.5±1.96ns 102.5 MElem/sec 1.00 73.9±1.95ns 103.2 MElem/sec
object_get_by_name/128 1.01 111.5±0.65ns ? ?/sec 1.00 110.2±1.09ns ? ?/sec
object_get_by_name/32 1.01 73.3±0.84ns ? ?/sec 1.00 72.6±1.04ns ? ?/sec
object_get_by_name/8 1.01 60.1±0.74ns ? ?/sec 1.00 59.7±0.90ns ? ?/sec
object_iter/128 1.00 2.6±0.03µs 46.7 MElem/sec 1.00 2.6±0.05µs 46.6 MElem/sec
object_iter/32 1.00 657.0±6.94ns 46.4 MElem/sec 1.00 658.1±11.33ns 46.4 MElem/sec
object_iter/8 1.00 169.5±1.86ns 45.0 MElem/sec 1.00 168.9±2.37ns 45.2 MElem/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 14.5 MiB |
| Avg memory | 13.0 MiB |
| CPU user | 150.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 15.2 MiB |
| Avg memory | 13.6 MiB |
| CPU user | 150.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733452885
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733354689)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733354689-2442-tbdrf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733521976
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733354689)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.03 21.5±0.73ms ? ?/sec 1.00 20.9±0.24ms ? ?/sec
bench_object_field_names_reverse_order 1.00 13.0±1.09ms ? ?/sec 1.04 13.5±0.45ms ? ?/sec
bench_object_list_partially_same_schema 1.00 1059.8±13.84µs ? ?/sec 1.00 1057.5±10.40µs ? ?/sec
bench_object_list_same_schema 1.00 21.3±0.13ms ? ?/sec 1.00 21.4±0.19ms ? ?/sec
bench_object_list_unknown_schema 1.01 10.9±0.25ms ? ?/sec 1.00 10.8±0.08ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.02ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.03 33.4±0.07ms ? ?/sec 1.00 32.4±0.59ms ? ?/sec
bench_object_unknown_schema 1.00 13.1±0.03ms ? ?/sec 1.00 13.2±0.05ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.00 39.8±0.47µs ? ?/sec 1.00 40.0±0.36µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.08µs ? ?/sec 1.00 4.3±0.08µs ? ?/sec
validation/validated_construction 1.02 68.8±1.71µs ? ?/sec 1.00 67.6±1.10µs ? ?/sec
validation/validation_cost 1.00 57.1±0.80µs ? ?/sec 1.00 57.1±0.89µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.9 MiB |
| CPU user | 132.3s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.7 MiB |
| CPU user | 132.8s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733473236
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733352952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_builder
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
bench_extend_metadata_builder 1.00 21.0±0.41ms ? ?/sec 1.01 21.3±0.36ms ? ?/sec
bench_object_field_names_reverse_order 1.00 12.9±0.78ms ? ?/sec 1.08 13.9±0.66ms ? ?/sec
bench_object_list_partially_same_schema 1.01 1065.0±14.32µs ? ?/sec 1.00 1057.5±9.75µs ? ?/sec
bench_object_list_same_schema 1.00 21.4±0.11ms ? ?/sec 1.00 21.4±0.11ms ? ?/sec
bench_object_list_unknown_schema 1.01 10.8±0.09ms ? ?/sec 1.00 10.8±0.05ms ? ?/sec
bench_object_partially_same_schema 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
bench_object_same_schema 1.05 33.5±0.06ms ? ?/sec 1.00 32.0±0.08ms ? ?/sec
bench_object_unknown_schema 1.00 13.2±0.04ms ? ?/sec 1.00 13.1±0.05ms ? ?/sec
iteration/unvalidated_fallible_iteration 1.00 2.8±0.00ms ? ?/sec 1.00 2.8±0.00ms ? ?/sec
iteration/validated_iteration 1.00 39.8±0.38µs ? ?/sec 1.01 40.2±0.34µs ? ?/sec
validation/unvalidated_construction 1.00 4.3±0.08µs ? ?/sec 1.01 4.3±0.08µs ? ?/sec
validation/validated_construction 1.00 67.4±1.06µs ? ?/sec 1.00 67.5±0.93µs ? ?/sec
validation/validation_cost 1.00 57.1±0.77µs ? ?/sec 1.00 57.1±0.91µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 17.4 MiB |
| CPU user | 132.9s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 54.1 MiB |
| Avg memory | 16.6 MiB |
| CPU user | 132.8s |
| CPU sys | 1.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733476472
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733355457)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733355457-2444-ktxgc 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_validation
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_validation
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733567163
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733357732)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733357732-2449-gpbs5 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733626848
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733358572)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing friendlymatthew/optimize-variant-object-path-access (16d7788ebec5d8c06bdce5e925b8a937d5fe35f5) to 1a6630b (merge-base) [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..16d7788ebec5d8c06bdce5e925b8a937d5fe35f5)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.04 20.0±0.45ms ? ?/sec 1.00 19.2±0.14ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.01 147.8±0.82ms ? ?/sec 1.00 146.9±0.25ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.00 309.2±1.97ms ? ?/sec 1.00 308.1±1.45ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 220.0±1.30ms ? ?/sec 1.00 219.7±1.22ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.01 6.4±0.08ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
variant_array_builder_build_262k_small_values 1.02 2.7±0.04ms ? ?/sec 1.00 2.6±0.05ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 224.8 MiB |
| CPU user | 131.3s |
| CPU sys | 5.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 227.2 MiB |
| CPU user | 131.6s |
| CPU sys | 4.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733639562
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733359404)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group friendlymatthew_optimize-variant-object-path-access main
----- --------------------------------------------------- ----
batch_json_string_to_variant json_list 8k string 1.02 19.7±0.17ms ? ?/sec 1.00 19.2±0.17ms ? ?/sec
batch_json_string_to_variant object - 1 depth(100 fields) random_json(2436 bytes per document) 1.01 147.6±0.78ms ? ?/sec 1.00 146.7±0.43ms ? ?/sec
batch_json_string_to_variant object - 1 depth(200 fields) random_json(4871 bytes per document) 1.01 308.8±1.65ms ? ?/sec 1.00 306.8±0.84ms ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document) 1.00 219.4±0.78ms ? ?/sec 1.00 220.0±1.43ms ? ?/sec
batch_json_string_to_variant repeated_struct 8k string 1.00 6.4±0.04ms ? ?/sec 1.00 6.4±0.05ms ? ?/sec
variant_array_builder_build_262k_small_values 1.02 2.7±0.10ms ? ?/sec 1.00 2.7±0.11ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 446.3 MiB |
| Avg memory | 225.9 MiB |
| CPU user | 131.5s |
| CPU sys | 5.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 444.9 MiB |
| Avg memory | 227.9 MiB |
| CPU user | 130.7s |
| CPU sys | 4.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733640386
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10358#issuecomment-5733360851)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5733360851-2453-dq2kg 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 1a6630bc2c198955a9d24b34ca560dcbb9167a7f (1a6630bc2c198955a9d24b34ca560dcbb9167a7f) to 1a6630bc2c198955a9d24b34ca560dcbb9167a7f [diff](https://github.com/apache/arrow-rs/compare/1a6630bc2c198955a9d24b34ca560dcbb9167a7f..1a6630bc2c198955a9d24b34ca560dcbb9167a7f)
<details><summary>Run configuration</summary>
```yaml
run benchmark variant_kernels
env:
BENCH_FILTER: "^(variant_array_builder|batch_json)"
baseline:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
changed:
ref: "1a6630bc2c198955a9d24b34ca560dcbb9167a7f"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: optimize variant object path access (79% lower latency) [arrow-rs]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #10358:
URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-6086383082
We ran some benchmarks against this PR using arrays where each row has its own metadata dictionary, which is what `json_to_variant` and `VariantArrayBuilder` produce. In that case `variant_get` with an object path is about 2–4x slower than on `main`. The wider the dictionary, the larger the slowdown.
From what we can see, there are three reasons:
1. **The cache misses on almost every row.** It's keyed on the metadata pointer, and with per-row metadata every row has a different pointer, even when the bytes are identical.
2. **Every miss fully validates the metadata, and does it twice.** It calls `VariantMetadata::try_new` once in the `variant_get` loop and again in `VariantPathRowBuilder`. `main` doesn't fully validate the dictionary on this path, so the cost of that check across all names lands on every row.
3. **Unsorted dictionaries still pay for path resolution.** Each miss runs a linear `get_entry` scan and allocates a `String`, then falls back to a by-name lookup anyway.
The existing `variant_get_unshredded_object_path_262k_rows` benchmark shares one metadata buffer across all rows, so the cache always hits and this case doesn't show up there. A benchmark with per-row metadata, for example `json_to_variant` over objects with varied key sets, would cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org