You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "MassivePizza (via GitHub)" <gi...@apache.org> on 2026/07/17 10:54:12 UTC
[PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
MassivePizza opened a new pull request, #10364:
URL: https://github.com/apache/arrow-rs/pull/10364
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
N/A
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Converting decimals and FixedLenByteArrays to individual allocs requires a lot of alloc+drop time and memory overhead.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Allocate one big buffer, and cut it up into slices of `bytes::Bytes`. This amortizes the alloc upfront, and effectively eliminates the drop overhead by turning a dealloc to an Arc decrement per slice.
This still has moderate memory overhead. `Bytes` is 32 bytes, while the slices usually expose less than 16 bytes at a time e.g. UUIDs or decimal128, but a lot more effort would be required to implement a (perhaps borrowed?) alternative that behaves like `FixedLenByteArray`.
There are a few other ideas worth exploring, primarily custom `ColumnValueEncoder`s that converts on the fly to avoid the temporary allocs all together.
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
Should be covered by existing tests. There are also new benches included:
<details>
<summary>decimal bench</summary>
```
Benchmarking decimal/default: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.4s, or reduce sample count to 90.
decimal/default time: [53.676 ms 53.979 ms 54.306 ms]
thrpt: [1.0790 GiB/s 1.0855 GiB/s 1.0916 GiB/s]
change:
time: [−44.611% −44.247% −43.885%] (p = 0.00 < 0.05)
thrpt: [+78.204% +79.362% +80.542%]
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
Benchmarking decimal/bloom_filter: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.8s, or reduce sample count to 70.
decimal/bloom_filter time: [66.597 ms 66.838 ms 67.095 ms]
thrpt: [894.26 MiB/s 897.70 MiB/s 900.94 MiB/s]
change:
time: [−40.229% −39.907% −39.555%] (p = 0.00 < 0.05)
thrpt: [+65.440% +66.408% +67.306%]
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
8 (8.00%) high mild
2 (2.00%) high severe
Benchmarking decimal/parquet_2: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 10.8s, or reduce sample count to 40.
decimal/parquet_2 time: [105.82 ms 106.23 ms 106.66 ms]
thrpt: [562.52 MiB/s 564.84 MiB/s 567.02 MiB/s]
change:
time: [−48.804% −48.523% −48.229%] (p = 0.00 < 0.05)
thrpt: [+93.160% +94.263% +95.329%]
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
Benchmarking decimal/zstd: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.4s, or reduce sample count to 70.
decimal/zstd time: [61.806 ms 62.013 ms 62.282 ms]
thrpt: [963.36 MiB/s 967.55 MiB/s 970.78 MiB/s]
change:
time: [−60.257% −59.907% −59.556%] (p = 0.00 < 0.05)
thrpt: [+147.25% +149.42% +151.62%]
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
1 (1.00%) low severe
2 (2.00%) low mild
5 (5.00%) high mild
3 (3.00%) high severe
Benchmarking decimal/zstd_parquet_2: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.9s, or reduce sample count to 40.
decimal/zstd_parquet_2 time: [119.51 ms 119.87 ms 120.29 ms]
thrpt: [498.82 MiB/s 500.54 MiB/s 502.07 MiB/s]
change:
time: [−55.716% −55.352% −54.992%] (p = 0.00 < 0.05)
thrpt: [+122.18% +123.98% +125.82%]
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
2 (2.00%) low mild
2 (2.00%) high mild
7 (7.00%) high severe
Benchmarking decimal/cdc: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.8s, or reduce sample count to 60.
decimal/cdc time: [77.810 ms 78.092 ms 78.389 ms]
thrpt: [765.42 MiB/s 768.33 MiB/s 771.11 MiB/s]
change:
time: [−35.689% −35.360% −35.011%] (p = 0.00 < 0.05)
thrpt: [+53.873% +54.703% +55.494%]
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high mild
```
</details>
<details>
<summary>fsb bench</summary>
```
fsb/default time: [5.4152 ms 5.4276 ms 5.4407 ms]
thrpt: [2.8943 GiB/s 2.9013 GiB/s 2.9079 GiB/s]
change:
time: [−27.824% −27.512% −27.215%] (p = 0.00 < 0.05)
thrpt: [+37.391% +37.953% +38.550%]
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) low mild
5 (5.00%) high mild
1 (1.00%) high severe
fsb/bloom_filter time: [5.8681 ms 5.8908 ms 5.9154 ms]
thrpt: [2.6621 GiB/s 2.6732 GiB/s 2.6835 GiB/s]
change:
time: [−25.701% −25.298% −24.896%] (p = 0.00 < 0.05)
thrpt: [+33.148% +33.866% +34.592%]
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe
fsb/parquet_2 time: [8.6936 ms 8.7692 ms 8.8503 ms]
thrpt: [1.7793 GiB/s 1.7957 GiB/s 1.8113 GiB/s]
change:
time: [−31.621% −31.005% −30.286%] (p = 0.00 < 0.05)
thrpt: [+43.444% +44.939% +46.243%]
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
9 (9.00%) high mild
1 (1.00%) high severe
fsb/zstd time: [7.1760 ms 7.2043 ms 7.2360 ms]
thrpt: [2.1762 GiB/s 2.1858 GiB/s 2.1944 GiB/s]
change:
time: [−21.480% −21.068% −20.631%] (p = 0.00 < 0.05)
thrpt: [+25.994% +26.692% +27.356%]
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
fsb/zstd_parquet_2 time: [9.2977 ms 9.4085 ms 9.5422 ms]
thrpt: [1.6503 GiB/s 1.6737 GiB/s 1.6937 GiB/s]
change:
time: [−30.266% −29.333% −28.383%] (p = 0.00 < 0.05)
thrpt: [+39.632% +41.508% +43.401%]
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
fsb/cdc time: [7.9524 ms 7.9700 ms 7.9878 ms]
thrpt: [1.9714 GiB/s 1.9758 GiB/s 1.9802 GiB/s]
change:
time: [−21.280% −20.793% −20.375%] (p = 0.00 < 0.05)
thrpt: [+25.589% +26.251% +27.033%]
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mildaa
```
</details>
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5049499229
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10364#issuecomment-5048483940)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main parquet-contiguous-decimal
----- ---- --------------------------
bool/bloom_filter 1.00 12.9±0.06ms 19.4 MB/sec 1.01 13.0±0.07ms 19.2 MB/sec
bool/cdc 1.00 15.6±0.05ms 16.1 MB/sec 1.01 15.7±0.06ms 15.9 MB/sec
bool/default 1.00 10.8±0.06ms 23.2 MB/sec 1.01 10.9±0.05ms 23.0 MB/sec
bool/parquet_2 1.00 14.6±0.07ms 17.1 MB/sec 1.01 14.7±0.06ms 17.0 MB/sec
bool/zstd 1.00 11.3±0.07ms 22.2 MB/sec 1.01 11.4±0.05ms 21.9 MB/sec
bool/zstd_parquet_2 1.00 15.0±0.07ms 16.7 MB/sec 1.01 15.1±0.06ms 16.6 MB/sec
bool_non_null/bloom_filter 1.00 7.0±0.03ms 18.0 MB/sec 1.00 6.9±0.02ms 18.1 MB/sec
bool_non_null/cdc 1.00 6.7±0.03ms 18.7 MB/sec 1.00 6.7±0.04ms 18.7 MB/sec
bool_non_null/default 1.00 4.2±0.02ms 30.1 MB/sec 1.00 4.2±0.02ms 30.1 MB/sec
bool_non_null/parquet_2 1.00 9.0±0.03ms 13.9 MB/sec 1.01 9.0±0.04ms 13.9 MB/sec
bool_non_null/zstd 1.00 4.5±0.02ms 27.7 MB/sec 1.00 4.5±0.02ms 27.7 MB/sec
bool_non_null/zstd_parquet_2 1.00 9.4±0.04ms 13.3 MB/sec 1.00 9.4±0.04ms 13.3 MB/sec
bool_ree/bloom_filter 1.00 30.5±0.06ms 7.0 MB/sec 1.01 30.7±0.08ms 6.9 MB/sec
bool_ree/cdc 1.00 32.0±0.16ms 6.6 MB/sec 1.00 31.9±0.10ms 6.7 MB/sec
bool_ree/default 1.00 27.8±0.07ms 7.7 MB/sec 1.01 27.9±0.09ms 7.6 MB/sec
bool_ree/parquet_2 1.00 29.3±0.07ms 7.3 MB/sec 1.01 29.5±0.10ms 7.2 MB/sec
bool_ree/zstd 1.00 28.6±0.15ms 7.4 MB/sec 1.00 28.6±0.11ms 7.4 MB/sec
bool_ree/zstd_parquet_2 1.00 30.0±0.13ms 7.1 MB/sec 1.00 29.8±0.09ms 7.1 MB/sec
decimal/bloom_filter 1.17 145.0±0.63ms 413.9 MB/sec 1.00 123.5±0.30ms 485.8 MB/sec
decimal/cdc 1.25 131.6±0.20ms 455.8 MB/sec 1.00 105.3±0.34ms 569.6 MB/sec
decimal/default 1.29 106.7±0.44ms 562.2 MB/sec 1.00 83.0±0.27ms 722.7 MB/sec
decimal/parquet_2 1.51 271.8±1.22ms 220.8 MB/sec 1.00 179.8±1.65ms 333.7 MB/sec
decimal/zstd 1.25 119.3±0.30ms 503.1 MB/sec 1.00 95.7±1.04ms 626.6 MB/sec
decimal/zstd_parquet_2 1.45 292.0±1.30ms 205.5 MB/sec 1.00 202.1±1.85ms 297.0 MB/sec
fixed_size_binary_ree/bloom_filter 1.20 69.8±0.31ms 14.4 MB/sec 1.00 58.2±0.26ms 17.3 MB/sec
fixed_size_binary_ree/cdc 1.18 72.8±0.26ms 13.8 MB/sec 1.00 61.7±0.21ms 16.3 MB/sec
fixed_size_binary_ree/default 1.21 62.9±0.25ms 16.0 MB/sec 1.00 51.8±0.22ms 19.5 MB/sec
fixed_size_binary_ree/parquet_2 1.23 79.6±0.29ms 12.6 MB/sec 1.00 64.7±0.20ms 15.6 MB/sec
fixed_size_binary_ree/zstd 1.20 68.3±0.28ms 14.8 MB/sec 1.00 57.1±0.18ms 17.6 MB/sec
fixed_size_binary_ree/zstd_parquet_2 1.22 79.9±0.25ms 12.6 MB/sec 1.00 65.3±0.22ms 15.4 MB/sec
float_with_nans/bloom_filter 1.17 91.9±0.27ms 152.3 MB/sec 1.00 78.8±0.23ms 177.6 MB/sec
float_with_nans/cdc 1.17 80.8±0.15ms 173.3 MB/sec 1.00 68.9±0.22ms 203.2 MB/sec
float_with_nans/default 1.19 73.4±0.16ms 190.8 MB/sec 1.00 61.6±0.20ms 227.3 MB/sec
float_with_nans/parquet_2 1.21 93.6±0.23ms 149.5 MB/sec 1.00 77.5±0.21ms 180.6 MB/sec
float_with_nans/zstd 1.12 111.3±0.17ms 125.8 MB/sec 1.00 99.4±0.20ms 140.8 MB/sec
float_with_nans/zstd_parquet_2 1.14 130.9±0.20ms 106.9 MB/sec 1.00 114.4±0.98ms 122.4 MB/sec
fsb/bloom_filter 1.16 9.4±0.05ms 1721.2 MB/sec 1.00 8.1±0.03ms 1992.5 MB/sec
fsb/cdc 1.13 11.7±0.04ms 1374.5 MB/sec 1.00 10.4±0.03ms 1547.3 MB/sec
fsb/default 1.19 8.3±0.04ms 1943.7 MB/sec 1.00 7.0±0.03ms 2.3 GB/sec
fsb/parquet_2 1.36 17.9±0.07ms 901.8 MB/sec 1.00 13.2±0.07ms 1226.2 MB/sec
fsb/zstd 1.14 10.8±0.06ms 1496.0 MB/sec 1.00 9.5±0.03ms 1699.5 MB/sec
fsb/zstd_parquet_2 1.34 18.8±0.06ms 856.6 MB/sec 1.00 14.0±0.06ms 1149.6 MB/sec
int32_ree/bloom_filter 1.01 40.5±0.26ms 10.0 MB/sec 1.00 40.0±0.11ms 10.2 MB/sec
int32_ree/cdc 1.01 40.4±0.16ms 10.1 MB/sec 1.00 40.1±0.09ms 10.1 MB/sec
int32_ree/default 1.01 36.7±0.15ms 11.1 MB/sec 1.00 36.3±0.09ms 11.2 MB/sec
int32_ree/parquet_2 1.01 36.7±0.16ms 11.1 MB/sec 1.00 36.2±0.08ms 11.2 MB/sec
int32_ree/zstd 1.01 37.2±0.16ms 10.9 MB/sec 1.00 36.8±0.12ms 11.1 MB/sec
int32_ree/zstd_parquet_2 1.01 37.1±0.16ms 11.0 MB/sec 1.00 36.8±0.10ms 11.1 MB/sec
int32_ree_95pct_null/bloom_filter 1.00 21.9±0.03ms 18.6 MB/sec 1.00 21.9±0.02ms 18.6 MB/sec
int32_ree_95pct_null/cdc 1.00 23.8±0.03ms 17.1 MB/sec 1.01 24.1±0.03ms 16.8 MB/sec
int32_ree_95pct_null/default 1.00 21.6±0.02ms 18.9 MB/sec 1.00 21.6±0.01ms 18.9 MB/sec
int32_ree_95pct_null/parquet_2 1.00 21.6±0.02ms 18.9 MB/sec 1.00 21.7±0.03ms 18.8 MB/sec
int32_ree_95pct_null/zstd 1.00 21.8±0.02ms 18.6 MB/sec 1.00 21.8±0.02ms 18.6 MB/sec
int32_ree_95pct_null/zstd_parquet_2 1.00 21.7±0.02ms 18.7 MB/sec 1.00 21.7±0.02ms 18.7 MB/sec
large_string_non_null/bloom_filter 1.00 71.8±0.11ms 3.5 GB/sec 1.01 72.8±0.09ms 3.4 GB/sec
large_string_non_null/cdc 1.00 242.9±1.11ms 1053.8 MB/sec 1.01 244.9±1.26ms 1045.4 MB/sec
large_string_non_null/default 1.00 54.0±0.10ms 4.6 GB/sec 1.02 54.9±0.09ms 4.6 GB/sec
large_string_non_null/parquet_2 1.00 54.1±0.10ms 4.6 GB/sec 1.02 54.9±0.09ms 4.6 GB/sec
large_string_non_null/zstd 1.00 54.1±0.10ms 4.6 GB/sec 1.02 54.9±0.09ms 4.6 GB/sec
large_string_non_null/zstd_parquet_2 1.00 54.1±0.10ms 4.6 GB/sec 1.02 55.0±0.09ms 4.5 GB/sec
list_nested/bloom_filter 1.00 147.1±0.19ms 196.1 MB/sec 1.00 147.4±0.20ms 195.7 MB/sec
list_nested/cdc 1.00 164.1±0.15ms 175.9 MB/sec 1.00 164.0±0.18ms 176.0 MB/sec
list_nested/default 1.00 133.1±0.17ms 216.9 MB/sec 1.00 132.6±0.17ms 217.6 MB/sec
list_nested/parquet_2 1.00 146.4±0.19ms 197.0 MB/sec 1.00 145.9±0.19ms 197.8 MB/sec
list_nested/zstd 1.00 142.2±0.15ms 202.9 MB/sec 1.00 141.6±0.18ms 203.8 MB/sec
list_nested/zstd_parquet_2 1.00 159.4±0.16ms 181.0 MB/sec 1.00 159.4±0.43ms 181.0 MB/sec
list_primitive/bloom_filter 1.08 304.0±2.55ms 1794.0 MB/sec 1.00 280.9±0.72ms 1941.5 MB/sec
list_primitive/cdc 1.04 324.1±6.76ms 1682.8 MB/sec 1.00 312.1±1.39ms 1747.2 MB/sec
list_primitive/default 1.11 226.1±1.93ms 2.4 GB/sec 1.00 203.6±0.83ms 2.6 GB/sec
list_primitive/parquet_2 1.10 248.8±0.65ms 2.1 GB/sec 1.00 226.8±0.43ms 2.3 GB/sec
list_primitive/zstd 1.02 466.1±3.39ms 1170.0 MB/sec 1.00 456.4±1.29ms 1194.9 MB/sec
list_primitive/zstd_parquet_2 1.00 452.8±0.52ms 1204.4 MB/sec 1.00 452.6±0.24ms 1205.0 MB/sec
list_primitive_non_null/bloom_filter 1.09 396.1±11.92ms 1374.0 MB/sec 1.00 362.8±4.65ms 1499.9 MB/sec
list_primitive_non_null/cdc 1.00 396.0±7.96ms 1374.5 MB/sec 1.00 396.5±8.13ms 1372.5 MB/sec
list_primitive_non_null/default 1.16 270.1±10.83ms 2014.6 MB/sec 1.00 232.2±4.31ms 2.3 GB/sec
list_primitive_non_null/parquet_2 1.06 283.0±25.96ms 1922.8 MB/sec 1.00 267.0±0.52ms 2038.1 MB/sec
list_primitive_non_null/zstd 1.06 691.0±16.27ms 787.6 MB/sec 1.00 653.0±5.16ms 833.5 MB/sec
list_primitive_non_null/zstd_parquet_2 1.05 667.3±0.74ms 815.6 MB/sec 1.00 636.8±1.80ms 854.7 MB/sec
list_primitive_sparse_99pct_null/bloom_filter 1.00 12.6±0.03ms 2.9 GB/sec 1.00 12.6±0.03ms 2.9 GB/sec
list_primitive_sparse_99pct_null/cdc 1.00 24.0±0.07ms 1559.4 MB/sec 1.00 23.9±0.06ms 1561.4 MB/sec
list_primitive_sparse_99pct_null/default 1.00 12.3±0.02ms 3.0 GB/sec 1.00 12.3±0.02ms 3.0 GB/sec
list_primitive_sparse_99pct_null/parquet_2 1.00 12.3±0.02ms 3.0 GB/sec 1.00 12.3±0.03ms 3.0 GB/sec
list_primitive_sparse_99pct_null/zstd 1.00 14.1±0.03ms 2.6 GB/sec 1.00 14.1±0.03ms 2.6 GB/sec
list_primitive_sparse_99pct_null/zstd_parquet_2 1.00 12.4±0.03ms 2.9 GB/sec 1.00 12.5±0.03ms 2.9 GB/sec
list_struct_with_list/bloom_filter 1.00 283.0±0.41ms 161.1 MB/sec 1.00 283.0±0.54ms 161.1 MB/sec
list_struct_with_list/cdc 1.00 321.6±0.38ms 141.8 MB/sec 1.00 322.5±0.52ms 141.4 MB/sec
list_struct_with_list/default 1.00 257.3±0.39ms 177.2 MB/sec 1.00 257.3±0.56ms 177.2 MB/sec
list_struct_with_list/parquet_2 1.00 275.2±0.87ms 165.7 MB/sec 1.00 274.7±0.47ms 166.0 MB/sec
list_struct_with_list/zstd 1.00 278.8±0.40ms 163.5 MB/sec 1.00 278.8±0.55ms 163.6 MB/sec
list_struct_with_list/zstd_parquet_2 1.00 298.2±0.37ms 152.9 MB/sec 1.00 298.0±0.54ms 153.0 MB/sec
primitive/bloom_filter 1.00 146.0±0.50ms 307.4 MB/sec 1.02 148.9±0.46ms 301.4 MB/sec
primitive/cdc 1.00 156.0±0.60ms 287.7 MB/sec 1.01 157.9±0.65ms 284.1 MB/sec
primitive/default 1.00 115.7±0.39ms 388.0 MB/sec 1.01 116.9±0.33ms 384.0 MB/sec
primitive/parquet_2 1.00 130.6±0.34ms 343.6 MB/sec 1.01 132.0±0.33ms 340.1 MB/sec
primitive/zstd 1.00 145.6±0.51ms 308.1 MB/sec 1.01 146.6±0.40ms 306.2 MB/sec
primitive/zstd_parquet_2 1.00 163.8±0.32ms 274.0 MB/sec 1.01 165.2±0.33ms 271.6 MB/sec
primitive_all_null/bloom_filter 1.00 877.3±3.22µs 50.0 GB/sec 1.01 887.7±3.13µs 49.4 GB/sec
primitive_all_null/cdc 1.00 19.0±0.30ms 2.3 GB/sec 1.01 19.2±0.35ms 2.3 GB/sec
primitive_all_null/default 1.00 269.7±1.39µs 162.5 GB/sec 1.00 270.4±1.52µs 162.1 GB/sec
primitive_all_null/parquet_2 1.00 268.0±1.63µs 163.5 GB/sec 1.01 271.5±1.33µs 161.4 GB/sec
primitive_all_null/zstd 1.00 390.6±1.64µs 112.2 GB/sec 1.00 388.7±1.67µs 112.7 GB/sec
primitive_all_null/zstd_parquet_2 1.00 350.9±1.75µs 124.9 GB/sec 1.01 353.5±1.51µs 124.0 GB/sec
primitive_non_null/bloom_filter 1.00 103.3±0.35ms 426.0 MB/sec 1.02 105.7±0.27ms 416.3 MB/sec
primitive_non_null/cdc 1.00 88.1±0.25ms 499.6 MB/sec 1.01 88.6±0.24ms 496.8 MB/sec
primitive_non_null/default 1.00 65.5±0.18ms 672.1 MB/sec 1.01 65.9±0.18ms 668.1 MB/sec
primitive_non_null/parquet_2 1.00 86.9±0.18ms 506.3 MB/sec 1.00 87.3±0.15ms 504.0 MB/sec
primitive_non_null/zstd 1.00 96.0±0.21ms 458.1 MB/sec 1.01 96.6±0.16ms 455.5 MB/sec
primitive_non_null/zstd_parquet_2 1.00 120.6±0.21ms 364.7 MB/sec 1.01 122.0±0.36ms 360.8 MB/sec
primitive_sparse_99pct_null/bloom_filter 1.01 12.0±0.08ms 3.7 GB/sec 1.00 11.9±0.07ms 3.7 GB/sec
primitive_sparse_99pct_null/cdc 1.00 29.2±0.33ms 1537.0 MB/sec 1.03 30.1±0.27ms 1489.7 MB/sec
primitive_sparse_99pct_null/default 1.01 10.6±0.05ms 4.1 GB/sec 1.00 10.5±0.03ms 4.2 GB/sec
primitive_sparse_99pct_null/parquet_2 1.00 10.6±0.03ms 4.1 GB/sec 1.00 10.5±0.03ms 4.2 GB/sec
primitive_sparse_99pct_null/zstd 1.00 13.9±0.05ms 3.2 GB/sec 1.00 13.9±0.04ms 3.2 GB/sec
primitive_sparse_99pct_null/zstd_parquet_2 1.00 12.5±0.05ms 3.5 GB/sec 1.00 12.5±0.05ms 3.5 GB/sec
short_string_non_null/bloom_filter 1.02 27.0±0.08ms 444.4 MB/sec 1.00 26.5±0.08ms 453.1 MB/sec
short_string_non_null/cdc 1.00 19.4±0.05ms 619.3 MB/sec 1.00 19.4±0.06ms 617.4 MB/sec
short_string_non_null/default 1.00 15.3±0.05ms 786.4 MB/sec 1.00 15.3±0.06ms 785.8 MB/sec
short_string_non_null/parquet_2 1.00 26.2±0.05ms 457.5 MB/sec 1.00 26.3±0.06ms 455.9 MB/sec
short_string_non_null/zstd 1.00 35.2±0.10ms 340.9 MB/sec 1.00 35.2±0.10ms 340.7 MB/sec
short_string_non_null/zstd_parquet_2 1.00 29.2±0.06ms 411.5 MB/sec 1.00 29.3±0.06ms 410.1 MB/sec
string/bloom_filter 1.08 211.7±20.90ms 2.4 GB/sec 1.00 196.3±2.67ms 2.6 GB/sec
string/cdc 1.00 208.4±7.92ms 2.5 GB/sec 1.00 208.7±6.20ms 2.5 GB/sec
string/default 1.17 124.9±19.66ms 4.1 GB/sec 1.00 106.7±2.86ms 4.8 GB/sec
string/parquet_2 1.14 113.8±0.49ms 4.5 GB/sec 1.00 99.4±4.07ms 5.1 GB/sec
string/zstd 1.03 420.5±13.58ms 1246.7 MB/sec 1.00 409.7±1.36ms 1279.6 MB/sec
string/zstd_parquet_2 1.00 388.2±0.55ms 1350.5 MB/sec 1.01 391.2±0.52ms 1340.0 MB/sec
string_and_binary_view/bloom_filter 1.00 63.4±0.28ms 508.7 MB/sec 1.01 64.3±0.29ms 501.9 MB/sec
string_and_binary_view/cdc 1.00 58.4±0.18ms 552.0 MB/sec 1.01 59.1±0.20ms 546.0 MB/sec
string_and_binary_view/default 1.00 47.7±0.16ms 675.9 MB/sec 1.01 48.4±0.14ms 665.9 MB/sec
string_and_binary_view/parquet_2 1.00 59.6±0.18ms 541.4 MB/sec 1.01 60.3±0.23ms 534.7 MB/sec
string_and_binary_view/zstd 1.00 84.6±0.15ms 381.1 MB/sec 1.01 85.3±0.20ms 378.0 MB/sec
string_and_binary_view/zstd_parquet_2 1.00 74.0±0.16ms 436.1 MB/sec 1.01 74.3±0.20ms 433.9 MB/sec
string_dictionary/bloom_filter 1.00 83.7±0.33ms 3.1 GB/sec 1.03 86.5±0.58ms 3.0 GB/sec
string_dictionary/cdc 1.00 46.9±0.42ms 5.5 GB/sec 1.02 47.9±0.36ms 5.4 GB/sec
string_dictionary/default 1.00 43.3±0.19ms 6.0 GB/sec 1.01 43.8±0.23ms 5.9 GB/sec
string_dictionary/parquet_2 1.00 48.6±0.13ms 5.3 GB/sec 1.02 49.3±0.17ms 5.2 GB/sec
string_dictionary/zstd 1.00 204.6±0.29ms 1290.7 MB/sec 1.00 205.5±0.25ms 1285.5 MB/sec
string_dictionary/zstd_parquet_2 1.00 194.0±0.18ms 1361.5 MB/sec 1.01 195.3±0.16ms 1352.2 MB/sec
string_non_null/bloom_filter 1.00 233.2±16.64ms 2.2 GB/sec 1.00 232.4±14.96ms 2.2 GB/sec
string_non_null/cdc 1.01 255.4±10.00ms 2.0 GB/sec 1.00 253.6±6.23ms 2.0 GB/sec
string_non_null/default 1.08 126.4±21.97ms 4.0 GB/sec 1.00 116.6±13.35ms 4.4 GB/sec
string_non_null/parquet_2 1.00 123.4±4.13ms 4.1 GB/sec 1.06 130.7±6.23ms 3.9 GB/sec
string_non_null/zstd 1.01 535.0±2.18ms 979.4 MB/sec 1.00 530.4±3.82ms 988.0 MB/sec
string_non_null/zstd_parquet_2 1.00 496.9±4.48ms 1054.6 MB/sec 1.00 497.1±0.52ms 1054.2 MB/sec
string_ree/bloom_filter 1.01 154.8±0.60ms 106.0 MB/sec 1.00 152.7±0.84ms 107.4 MB/sec
string_ree/cdc 1.28 224.7±0.70ms 73.0 MB/sec 1.00 175.7±0.57ms 93.4 MB/sec
string_ree/default 1.01 127.0±1.04ms 129.2 MB/sec 1.00 125.4±1.04ms 130.9 MB/sec
string_ree/parquet_2 1.00 178.9±0.38ms 91.7 MB/sec 1.01 180.5±0.47ms 90.9 MB/sec
string_ree/zstd 1.01 180.6±0.49ms 90.9 MB/sec 1.00 179.3±0.62ms 91.5 MB/sec
string_ree/zstd_parquet_2 1.00 193.9±0.39ms 84.6 MB/sec 1.01 196.0±0.74ms 83.7 MB/sec
string_ree_95pct_null/bloom_filter 1.00 26.2±0.10ms 53.8 MB/sec 1.00 26.1±0.07ms 53.9 MB/sec
string_ree_95pct_null/cdc 1.00 32.2±0.09ms 43.6 MB/sec 1.01 32.5±0.08ms 43.2 MB/sec
string_ree_95pct_null/default 1.00 24.7±0.08ms 57.0 MB/sec 1.00 24.7±0.05ms 57.0 MB/sec
string_ree_95pct_null/parquet_2 1.00 24.7±0.08ms 57.0 MB/sec 1.00 24.7±0.04ms 57.0 MB/sec
string_ree_95pct_null/zstd 1.00 25.5±0.09ms 55.1 MB/sec 1.00 25.5±0.05ms 55.2 MB/sec
string_ree_95pct_null/zstd_parquet_2 1.00 25.4±0.08ms 55.3 MB/sec 1.00 25.4±0.06ms 55.3 MB/sec
struct_all_null/bloom_filter 1.01 374.1±1.69µs 42.1 GB/sec 1.00 371.8±1.53µs 42.4 GB/sec
struct_all_null/cdc 1.00 7.7±0.05ms 2.0 GB/sec 1.01 7.9±0.09ms 2.0 GB/sec
struct_all_null/default 1.00 116.2±0.63µs 135.6 GB/sec 1.00 115.8±0.65µs 136.0 GB/sec
struct_all_null/parquet_2 1.01 115.7±0.68µs 136.2 GB/sec 1.00 115.0±0.50µs 137.0 GB/sec
struct_all_null/zstd 1.00 167.6±0.67µs 94.0 GB/sec 1.00 166.9±0.66µs 94.3 GB/sec
struct_all_null/zstd_parquet_2 1.00 150.6±0.67µs 104.6 GB/sec 1.01 152.4±0.56µs 103.3 GB/sec
struct_non_null/bloom_filter 1.00 44.9±0.14ms 356.1 MB/sec 1.01 45.3±0.23ms 353.2 MB/sec
struct_non_null/cdc 1.00 44.6±0.14ms 359.0 MB/sec 1.00 44.7±0.15ms 357.9 MB/sec
struct_non_null/default 1.00 31.3±0.12ms 511.9 MB/sec 1.00 31.3±0.12ms 510.9 MB/sec
struct_non_null/parquet_2 1.00 40.0±0.09ms 400.3 MB/sec 1.03 41.0±1.51ms 390.4 MB/sec
struct_non_null/zstd 1.00 39.8±0.07ms 401.6 MB/sec 1.00 40.0±0.12ms 399.9 MB/sec
struct_non_null/zstd_parquet_2 1.00 53.8±0.12ms 297.2 MB/sec 1.00 54.1±0.13ms 296.0 MB/sec
struct_sparse_99pct_null/bloom_filter 1.00 6.4±0.02ms 2.5 GB/sec 1.06 6.8±0.02ms 2.3 GB/sec
struct_sparse_99pct_null/cdc 1.00 13.2±0.07ms 1220.2 MB/sec 1.08 14.2±0.08ms 1134.9 MB/sec
struct_sparse_99pct_null/default 1.00 5.9±0.01ms 2.7 GB/sec 1.06 6.3±0.01ms 2.5 GB/sec
struct_sparse_99pct_null/parquet_2 1.00 5.9±0.01ms 2.7 GB/sec 1.06 6.2±0.01ms 2.5 GB/sec
struct_sparse_99pct_null/zstd 1.00 7.3±0.02ms 2.2 GB/sec 1.05 7.6±0.01ms 2.1 GB/sec
struct_sparse_99pct_null/zstd_parquet_2 1.00 6.7±0.01ms 2.4 GB/sec 1.06 7.0±0.01ms 2.2 GB/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2990.7s |
| Peak memory | 2.8 GiB |
| Avg memory | 2.6 GiB |
| CPU user | 2879.0s |
| CPU sys | 108.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 2910.6s |
| Peak memory | 2.9 GiB |
| Avg memory | 2.7 GiB |
| CPU user | 2840.7s |
| CPU sys | 64.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5048483940
run benchmark arrow_writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "MassivePizza (via GitHub)" <gi...@apache.org>.
MassivePizza commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3615872563
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1834,14 +1834,15 @@ fn get_interval_ym_array_slice(
array: &arrow_array::IntervalYearMonthArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
+ const VALUE_SIZE: usize = 12;
+ let mut arena = Vec::with_capacity(indices.len() * VALUE_SIZE);
for i in indices {
- let mut value = array.value(i).to_le_bytes().to_vec();
- let mut suffix = vec![0; 8];
- value.append(&mut suffix);
- values.push(FixedLenByteArray::from(ByteArray::from(value)))
+ let mut out = [0; 12];
+ let value = array.value(i);
+ out[0..4].copy_from_slice(&value.to_le_bytes());
+ arena.extend_from_slice(&out);
Review Comment:
Looked at the codegen, and it avoided some XORs. The real benefit came from rewriting it to use `chunks_exact_mut` (see https://github.com/apache/arrow-rs/pull/10364/changes/baa13dd29c458595eb6913fc2a24885309cbf6c1).
And I preferred the per-value XOR over a zeroed alloc, since the chunk is being filled linearly anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5081002079
Thanks @MassivePizza and @Jefffrey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5048516123
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10364#issuecomment-5048483940)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5048483940-1227-vfmcg 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing parquet-contiguous-decimal (0495502d2d1a821d383ad7112b7e108aa84457a9) to cd17899 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/cd17899fe4c68c812a9585ccaf4a36f1d30e0978..0495502d2d1a821d383ad7112b7e108aa84457a9)
BENCH_NAME=arrow_writer
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3606350967
##########
parquet/src/column/writer/mod.rs:
##########
@@ -1649,12 +1649,16 @@ fn update_max<T: ParquetValueType>(descr: &ColumnDescriptor, val: &T, max: &mut
update_stat::<T, _>(descr, val, max, |cur| compare_greater(descr, val, cur))
}
-#[inline]
-#[allow(clippy::eq_op)]
fn is_nan<T: ParquetValueType>(descr: &ColumnDescriptor, val: &T) -> bool {
+ is_nan_with(descr.logical_type_ref(), val)
+}
+
+#[inline(always)]
+#[allow(clippy::eq_op)]
+fn is_nan_with<T: ParquetValueType>(logical_type_ref: Option<&LogicalType>, val: &T) -> bool {
Review Comment:
I don't think the compare_greater/is_nan changes are really relevant to this PR, and will conflict with work already in progress to revamp NaN handling per changes to the spec (#9619). We can revisit once that merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3607366931
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1834,14 +1834,15 @@ fn get_interval_ym_array_slice(
array: &arrow_array::IntervalYearMonthArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
+ const VALUE_SIZE: usize = 12;
+ let mut arena = Vec::with_capacity(indices.len() * VALUE_SIZE);
for i in indices {
- let mut value = array.value(i).to_le_bytes().to_vec();
- let mut suffix = vec![0; 8];
- value.append(&mut suffix);
- values.push(FixedLenByteArray::from(ByteArray::from(value)))
+ let mut out = [0; 12];
+ let value = array.value(i);
+ out[0..4].copy_from_slice(&value.to_le_bytes());
+ arena.extend_from_slice(&out);
Review Comment:
would it make a difference to change arena to `vec![0; indices.len() * VALUE_SIZE]` then use `copy_from_slice` into offsets to arena to avoid this intermediate `out`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5054050374
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10364#issuecomment-5053492576)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main parquet-contiguous-decimal
----- ---- --------------------------
bool/bloom_filter 1.00 12.8±0.05ms 19.5 MB/sec 1.03 13.3±0.06ms 18.9 MB/sec
bool/cdc 1.00 15.5±0.03ms 16.1 MB/sec 1.01 15.7±0.06ms 15.9 MB/sec
bool/default 1.00 10.7±0.04ms 23.3 MB/sec 1.04 11.1±0.06ms 22.5 MB/sec
bool/parquet_2 1.00 14.6±0.05ms 17.1 MB/sec 1.02 14.9±0.08ms 16.8 MB/sec
bool/zstd 1.00 11.3±0.04ms 22.2 MB/sec 1.03 11.7±0.08ms 21.5 MB/sec
bool/zstd_parquet_2 1.00 15.0±0.05ms 16.7 MB/sec 1.02 15.2±0.07ms 16.4 MB/sec
bool_non_null/bloom_filter 1.00 7.0±0.03ms 18.0 MB/sec 1.00 6.9±0.03ms 18.0 MB/sec
bool_non_null/cdc 1.00 6.7±0.04ms 18.7 MB/sec 1.02 6.8±0.03ms 18.4 MB/sec
bool_non_null/default 1.01 4.2±0.02ms 29.9 MB/sec 1.00 4.2±0.02ms 30.1 MB/sec
bool_non_null/parquet_2 1.01 9.0±0.04ms 13.9 MB/sec 1.00 8.9±0.04ms 14.0 MB/sec
bool_non_null/zstd 1.00 4.5±0.02ms 27.7 MB/sec 1.00 4.5±0.02ms 27.8 MB/sec
bool_non_null/zstd_parquet_2 1.00 9.4±0.04ms 13.3 MB/sec 1.00 9.4±0.04ms 13.3 MB/sec
bool_ree/bloom_filter 1.00 30.4±0.04ms 7.0 MB/sec 1.02 30.9±0.11ms 6.9 MB/sec
bool_ree/cdc 1.00 31.7±0.07ms 6.7 MB/sec 1.00 31.8±0.10ms 6.7 MB/sec
bool_ree/default 1.00 27.7±0.05ms 7.7 MB/sec 1.01 28.0±0.10ms 7.6 MB/sec
bool_ree/parquet_2 1.00 29.3±0.04ms 7.3 MB/sec 1.01 29.6±0.11ms 7.2 MB/sec
bool_ree/zstd 1.00 28.3±0.04ms 7.5 MB/sec 1.01 28.6±0.12ms 7.4 MB/sec
bool_ree/zstd_parquet_2 1.00 29.6±0.04ms 7.2 MB/sec 1.01 29.9±0.10ms 7.1 MB/sec
decimal/bloom_filter 1.19 149.4±0.74ms 401.5 MB/sec 1.00 125.5±2.04ms 477.9 MB/sec
decimal/cdc 1.25 132.6±0.39ms 452.6 MB/sec 1.00 105.8±0.41ms 567.0 MB/sec
decimal/default 1.29 107.2±0.47ms 559.9 MB/sec 1.00 83.2±0.25ms 721.3 MB/sec
decimal/parquet_2 1.51 273.7±1.24ms 219.2 MB/sec 1.00 181.6±2.11ms 330.3 MB/sec
decimal/zstd 1.25 120.0±0.72ms 500.1 MB/sec 1.00 95.9±1.00ms 625.4 MB/sec
decimal/zstd_parquet_2 1.44 294.1±1.05ms 204.0 MB/sec 1.00 203.7±2.26ms 294.5 MB/sec
fixed_size_binary_ree/bloom_filter 1.19 69.7±0.16ms 14.5 MB/sec 1.00 58.8±0.26ms 17.1 MB/sec
fixed_size_binary_ree/cdc 1.17 72.6±0.13ms 13.9 MB/sec 1.00 62.0±0.27ms 16.3 MB/sec
fixed_size_binary_ree/default 1.20 62.6±0.13ms 16.1 MB/sec 1.00 52.0±0.22ms 19.4 MB/sec
fixed_size_binary_ree/parquet_2 1.22 79.2±0.19ms 12.7 MB/sec 1.00 64.9±0.17ms 15.5 MB/sec
fixed_size_binary_ree/zstd 1.17 68.0±0.14ms 14.8 MB/sec 1.00 58.0±1.52ms 17.4 MB/sec
fixed_size_binary_ree/zstd_parquet_2 1.21 79.9±0.17ms 12.6 MB/sec 1.00 66.0±1.81ms 15.3 MB/sec
float_with_nans/bloom_filter 1.17 93.2±0.30ms 150.3 MB/sec 1.00 79.7±0.20ms 175.6 MB/sec
float_with_nans/cdc 1.17 81.4±0.19ms 172.1 MB/sec 1.00 69.6±0.25ms 201.1 MB/sec
float_with_nans/default 1.19 73.7±0.13ms 189.9 MB/sec 1.00 61.9±0.16ms 226.3 MB/sec
float_with_nans/parquet_2 1.22 94.8±0.21ms 147.6 MB/sec 1.00 77.8±0.19ms 179.9 MB/sec
float_with_nans/zstd 1.12 111.8±0.13ms 125.2 MB/sec 1.00 99.6±0.18ms 140.6 MB/sec
float_with_nans/zstd_parquet_2 1.15 132.2±0.20ms 105.9 MB/sec 1.00 114.6±0.17ms 122.2 MB/sec
fsb/bloom_filter 1.26 10.3±0.06ms 1565.4 MB/sec 1.00 8.2±0.06ms 1978.2 MB/sec
fsb/cdc 1.19 12.5±0.04ms 1286.3 MB/sec 1.00 10.5±0.05ms 1531.7 MB/sec
fsb/default 1.25 8.8±0.03ms 1834.0 MB/sec 1.00 7.0±0.05ms 2.2 GB/sec
fsb/parquet_2 1.49 19.7±0.21ms 816.7 MB/sec 1.00 13.3±0.10ms 1215.3 MB/sec
fsb/zstd 1.18 11.3±0.04ms 1423.7 MB/sec 1.00 9.6±0.03ms 1682.2 MB/sec
fsb/zstd_parquet_2 1.43 20.4±0.21ms 789.2 MB/sec 1.00 14.3±0.15ms 1129.9 MB/sec
int32_ree/bloom_filter 1.00 40.6±0.28ms 10.0 MB/sec 1.00 40.7±0.18ms 10.0 MB/sec
int32_ree/cdc 1.00 40.0±0.04ms 10.2 MB/sec 1.01 40.5±0.10ms 10.0 MB/sec
int32_ree/default 1.00 36.5±0.23ms 11.1 MB/sec 1.01 36.7±0.11ms 11.1 MB/sec
int32_ree/parquet_2 1.00 36.5±0.21ms 11.1 MB/sec 1.01 36.7±0.10ms 11.1 MB/sec
int32_ree/zstd 1.00 37.0±0.23ms 11.0 MB/sec 1.01 37.3±0.09ms 10.9 MB/sec
int32_ree/zstd_parquet_2 1.00 37.0±0.21ms 11.0 MB/sec 1.00 37.2±0.12ms 10.9 MB/sec
int32_ree_95pct_null/bloom_filter 1.00 21.9±0.02ms 18.6 MB/sec 1.00 21.9±0.04ms 18.5 MB/sec
int32_ree_95pct_null/cdc 1.00 23.9±0.04ms 17.1 MB/sec 1.02 24.2±0.07ms 16.8 MB/sec
int32_ree_95pct_null/default 1.00 21.6±0.02ms 18.9 MB/sec 1.00 21.6±0.01ms 18.8 MB/sec
int32_ree_95pct_null/parquet_2 1.00 21.6±0.02ms 18.9 MB/sec 1.00 21.6±0.02ms 18.8 MB/sec
int32_ree_95pct_null/zstd 1.00 21.8±0.02ms 18.6 MB/sec 1.00 21.9±0.02ms 18.6 MB/sec
int32_ree_95pct_null/zstd_parquet_2 1.00 21.8±0.02ms 18.7 MB/sec 1.00 21.8±0.02ms 18.7 MB/sec
large_string_non_null/bloom_filter 1.03 76.1±0.50ms 3.3 GB/sec 1.00 74.0±1.15ms 3.4 GB/sec
large_string_non_null/cdc 1.00 243.9±1.17ms 1049.7 MB/sec 1.00 244.1±1.09ms 1048.7 MB/sec
large_string_non_null/default 1.07 58.3±0.69ms 4.3 GB/sec 1.00 54.6±0.10ms 4.6 GB/sec
large_string_non_null/parquet_2 1.02 58.4±0.58ms 4.3 GB/sec 1.00 57.2±0.75ms 4.4 GB/sec
large_string_non_null/zstd 1.03 58.5±0.46ms 4.3 GB/sec 1.00 57.1±0.83ms 4.4 GB/sec
large_string_non_null/zstd_parquet_2 1.07 58.7±0.33ms 4.3 GB/sec 1.00 54.7±0.10ms 4.6 GB/sec
list_nested/bloom_filter 1.00 149.3±1.14ms 193.2 MB/sec 1.00 148.7±0.61ms 194.1 MB/sec
list_nested/cdc 1.00 165.3±0.54ms 174.6 MB/sec 1.00 165.3±0.28ms 174.6 MB/sec
list_nested/default 1.01 134.5±0.32ms 214.5 MB/sec 1.00 133.5±0.37ms 216.1 MB/sec
list_nested/parquet_2 1.00 146.7±0.16ms 196.7 MB/sec 1.00 147.2±0.31ms 196.0 MB/sec
list_nested/zstd 1.00 142.7±0.19ms 202.2 MB/sec 1.00 143.2±0.32ms 201.5 MB/sec
list_nested/zstd_parquet_2 1.00 159.8±0.18ms 180.6 MB/sec 1.00 160.2±0.33ms 180.1 MB/sec
list_primitive/bloom_filter 1.01 288.5±1.61ms 1890.6 MB/sec 1.00 285.4±0.60ms 1910.7 MB/sec
list_primitive/cdc 1.02 321.3±1.13ms 1697.2 MB/sec 1.00 314.5±0.89ms 1734.3 MB/sec
list_primitive/default 1.00 206.9±1.26ms 2.6 GB/sec 1.00 206.0±0.66ms 2.6 GB/sec
list_primitive/parquet_2 1.00 229.2±2.92ms 2.3 GB/sec 1.00 228.5±0.32ms 2.3 GB/sec
list_primitive/zstd 1.00 460.5±1.76ms 1184.2 MB/sec 1.00 458.6±0.59ms 1189.3 MB/sec
list_primitive/zstd_parquet_2 1.00 454.9±0.39ms 1198.8 MB/sec 1.00 454.4±0.31ms 1200.1 MB/sec
list_primitive_non_null/bloom_filter 1.10 409.5±4.85ms 1329.1 MB/sec 1.00 371.2±8.11ms 1466.1 MB/sec
list_primitive_non_null/cdc 1.01 400.9±8.20ms 1357.7 MB/sec 1.00 396.0±8.12ms 1374.2 MB/sec
list_primitive_non_null/default 1.06 255.7±3.66ms 2.1 GB/sec 1.00 240.9±2.63ms 2.2 GB/sec
list_primitive_non_null/parquet_2 1.00 273.1±14.54ms 1993.0 MB/sec 1.01 276.6±0.74ms 1967.6 MB/sec
list_primitive_non_null/zstd 1.03 683.4±11.58ms 796.4 MB/sec 1.00 666.5±5.39ms 816.6 MB/sec
list_primitive_non_null/zstd_parquet_2 1.01 641.7±4.57ms 848.1 MB/sec 1.00 636.9±1.16ms 854.5 MB/sec
list_primitive_sparse_99pct_null/bloom_filter 1.05 13.4±0.03ms 2.7 GB/sec 1.00 12.7±0.04ms 2.9 GB/sec
list_primitive_sparse_99pct_null/cdc 1.03 25.0±0.05ms 1496.9 MB/sec 1.00 24.1±0.08ms 1548.7 MB/sec
list_primitive_sparse_99pct_null/default 1.05 12.9±0.03ms 2.8 GB/sec 1.00 12.3±0.04ms 3.0 GB/sec
list_primitive_sparse_99pct_null/parquet_2 1.04 12.9±0.04ms 2.8 GB/sec 1.00 12.4±0.05ms 3.0 GB/sec
list_primitive_sparse_99pct_null/zstd 1.04 14.8±0.05ms 2.5 GB/sec 1.00 14.2±0.04ms 2.6 GB/sec
list_primitive_sparse_99pct_null/zstd_parquet_2 1.04 13.1±0.06ms 2.8 GB/sec 1.00 12.6±0.04ms 2.9 GB/sec
list_struct_with_list/bloom_filter 1.01 291.4±0.71ms 156.5 MB/sec 1.00 287.7±0.74ms 158.5 MB/sec
list_struct_with_list/cdc 1.00 323.8±1.05ms 140.8 MB/sec 1.00 322.2±0.54ms 141.5 MB/sec
list_struct_with_list/default 1.01 261.0±0.67ms 174.7 MB/sec 1.00 259.3±0.73ms 175.8 MB/sec
list_struct_with_list/parquet_2 1.00 278.7±0.51ms 163.6 MB/sec 1.00 277.4±0.88ms 164.4 MB/sec
list_struct_with_list/zstd 1.00 282.3±1.20ms 161.5 MB/sec 1.00 281.1±1.00ms 162.2 MB/sec
list_struct_with_list/zstd_parquet_2 1.00 300.2±1.22ms 151.9 MB/sec 1.00 299.0±1.67ms 152.5 MB/sec
primitive/bloom_filter 1.00 146.6±0.57ms 306.0 MB/sec 1.04 152.5±0.55ms 294.3 MB/sec
primitive/cdc 1.00 156.3±0.87ms 287.0 MB/sec 1.03 160.8±0.67ms 279.1 MB/sec
primitive/default 1.00 115.6±0.23ms 388.2 MB/sec 1.03 119.6±0.35ms 375.3 MB/sec
primitive/parquet_2 1.00 130.9±0.48ms 342.7 MB/sec 1.03 134.7±0.34ms 333.2 MB/sec
primitive/zstd 1.00 145.2±0.38ms 309.0 MB/sec 1.03 149.3±0.42ms 300.7 MB/sec
primitive/zstd_parquet_2 1.00 164.1±0.49ms 273.5 MB/sec 1.02 167.8±0.40ms 267.4 MB/sec
primitive_all_null/bloom_filter 1.05 925.0±4.35µs 47.4 GB/sec 1.00 884.0±2.80µs 49.6 GB/sec
primitive_all_null/cdc 1.00 19.0±0.25ms 2.3 GB/sec 1.00 19.1±0.24ms 2.3 GB/sec
primitive_all_null/default 1.02 273.4±1.22µs 160.3 GB/sec 1.00 268.2±1.46µs 163.4 GB/sec
primitive_all_null/parquet_2 1.00 270.4±1.18µs 162.1 GB/sec 1.00 269.5±1.18µs 162.6 GB/sec
primitive_all_null/zstd 1.00 391.2±1.37µs 112.0 GB/sec 1.00 392.4±1.61µs 111.7 GB/sec
primitive_all_null/zstd_parquet_2 1.00 351.3±1.37µs 124.8 GB/sec 1.01 353.3±1.55µs 124.0 GB/sec
primitive_non_null/bloom_filter 1.00 104.0±0.37ms 422.9 MB/sec 1.03 107.6±0.52ms 408.8 MB/sec
primitive_non_null/cdc 1.00 88.4±0.33ms 497.7 MB/sec 1.02 89.9±0.30ms 489.3 MB/sec
primitive_non_null/default 1.00 65.7±0.19ms 670.1 MB/sec 1.03 67.3±0.34ms 653.7 MB/sec
primitive_non_null/parquet_2 1.00 87.5±0.23ms 502.9 MB/sec 1.02 89.0±0.20ms 494.6 MB/sec
primitive_non_null/zstd 1.00 96.2±0.17ms 457.4 MB/sec 1.02 98.2±0.24ms 447.9 MB/sec
primitive_non_null/zstd_parquet_2 1.00 120.7±0.16ms 364.4 MB/sec 1.02 122.7±0.25ms 358.7 MB/sec
primitive_sparse_99pct_null/bloom_filter 1.13 13.5±0.07ms 3.2 GB/sec 1.00 12.0±0.09ms 3.6 GB/sec
primitive_sparse_99pct_null/cdc 1.00 30.4±0.21ms 1475.2 MB/sec 1.00 30.5±0.28ms 1470.6 MB/sec
primitive_sparse_99pct_null/default 1.05 11.2±0.05ms 3.9 GB/sec 1.00 10.7±0.04ms 4.1 GB/sec
primitive_sparse_99pct_null/parquet_2 1.06 11.2±0.03ms 3.9 GB/sec 1.00 10.6±0.04ms 4.1 GB/sec
primitive_sparse_99pct_null/zstd 1.05 14.7±0.03ms 3.0 GB/sec 1.00 14.0±0.06ms 3.1 GB/sec
primitive_sparse_99pct_null/zstd_parquet_2 1.04 13.2±0.04ms 3.3 GB/sec 1.00 12.7±0.05ms 3.5 GB/sec
short_string_non_null/bloom_filter 1.00 26.4±0.09ms 454.1 MB/sec 1.00 26.4±0.22ms 455.1 MB/sec
short_string_non_null/cdc 1.00 19.6±0.06ms 612.7 MB/sec 1.00 19.6±0.06ms 611.2 MB/sec
short_string_non_null/default 1.00 15.2±0.05ms 786.9 MB/sec 1.01 15.4±0.11ms 781.7 MB/sec
short_string_non_null/parquet_2 1.00 26.2±0.06ms 457.5 MB/sec 1.01 26.5±0.10ms 452.9 MB/sec
short_string_non_null/zstd 1.00 35.2±0.12ms 340.7 MB/sec 1.00 35.3±0.13ms 340.3 MB/sec
short_string_non_null/zstd_parquet_2 1.00 29.2±0.10ms 411.1 MB/sec 1.01 29.6±0.06ms 405.2 MB/sec
string/bloom_filter 1.00 198.4±6.03ms 2.6 GB/sec 1.13 223.5±23.82ms 2.3 GB/sec
string/cdc 1.00 208.1±8.32ms 2.5 GB/sec 1.01 210.7±5.92ms 2.4 GB/sec
string/default 1.00 113.4±9.87ms 4.5 GB/sec 1.10 124.2±19.93ms 4.1 GB/sec
string/parquet_2 1.33 132.9±0.46ms 3.9 GB/sec 1.00 99.9±2.03ms 5.1 GB/sec
string/zstd 1.02 424.5±8.26ms 1235.0 MB/sec 1.00 418.2±2.20ms 1253.6 MB/sec
string/zstd_parquet_2 1.01 401.1±4.68ms 1307.0 MB/sec 1.00 395.5±2.48ms 1325.6 MB/sec
string_and_binary_view/bloom_filter 1.01 65.1±0.29ms 495.7 MB/sec 1.00 64.2±0.37ms 502.4 MB/sec
string_and_binary_view/cdc 1.01 59.4±0.20ms 542.7 MB/sec 1.00 58.9±0.23ms 547.5 MB/sec
string_and_binary_view/default 1.00 48.6±0.20ms 663.1 MB/sec 1.00 48.4±0.18ms 665.9 MB/sec
string_and_binary_view/parquet_2 1.00 60.3±0.26ms 534.4 MB/sec 1.00 60.2±0.21ms 535.3 MB/sec
string_and_binary_view/zstd 1.00 85.5±0.25ms 377.2 MB/sec 1.00 85.2±0.20ms 378.4 MB/sec
string_and_binary_view/zstd_parquet_2 1.00 74.5±0.27ms 432.7 MB/sec 1.00 74.4±0.33ms 433.5 MB/sec
string_dictionary/bloom_filter 1.02 87.6±0.54ms 2.9 GB/sec 1.00 85.5±0.29ms 3.0 GB/sec
string_dictionary/cdc 1.01 47.9±0.27ms 5.4 GB/sec 1.00 47.6±0.38ms 5.4 GB/sec
string_dictionary/default 1.02 44.1±0.58ms 5.9 GB/sec 1.00 43.4±0.48ms 5.9 GB/sec
string_dictionary/parquet_2 1.02 49.3±0.19ms 5.2 GB/sec 1.00 48.4±0.17ms 5.3 GB/sec
string_dictionary/zstd 1.00 206.0±1.43ms 1282.4 MB/sec 1.00 205.0±1.25ms 1288.7 MB/sec
string_dictionary/zstd_parquet_2 1.00 195.0±0.18ms 1354.8 MB/sec 1.00 195.0±0.76ms 1354.3 MB/sec
string_non_null/bloom_filter 1.07 247.9±11.50ms 2.1 GB/sec 1.00 232.4±14.26ms 2.2 GB/sec
string_non_null/cdc 1.01 259.5±9.69ms 2019.1 MB/sec 1.00 257.3±8.41ms 2036.5 MB/sec
string_non_null/default 1.07 124.9±10.48ms 4.1 GB/sec 1.00 116.9±12.91ms 4.4 GB/sec
string_non_null/parquet_2 1.00 127.2±9.40ms 4.0 GB/sec 1.00 127.3±6.03ms 4.0 GB/sec
string_non_null/zstd 1.00 542.3±4.20ms 966.2 MB/sec 1.00 541.2±3.91ms 968.2 MB/sec
string_non_null/zstd_parquet_2 1.00 497.4±0.44ms 1053.5 MB/sec 1.00 496.0±1.01ms 1056.4 MB/sec
string_ree/bloom_filter 1.00 123.0±1.62ms 133.4 MB/sec 1.00 123.2±1.35ms 133.1 MB/sec
string_ree/cdc 1.00 185.0±3.53ms 88.7 MB/sec 1.00 185.6±5.26ms 88.4 MB/sec
string_ree/default 1.00 94.4±1.58ms 173.8 MB/sec 1.31 123.6±1.08ms 132.7 MB/sec
string_ree/parquet_2 1.00 130.8±0.32ms 125.4 MB/sec 1.02 133.8±0.25ms 122.6 MB/sec
string_ree/zstd 1.00 131.9±1.49ms 124.4 MB/sec 1.00 132.4±0.52ms 124.0 MB/sec
string_ree/zstd_parquet_2 1.00 145.4±0.30ms 112.9 MB/sec 1.02 148.4±0.27ms 110.6 MB/sec
string_ree_95pct_null/bloom_filter 1.00 26.2±0.08ms 53.6 MB/sec 1.01 26.4±0.11ms 53.3 MB/sec
string_ree_95pct_null/cdc 1.00 32.5±0.08ms 43.3 MB/sec 1.01 32.8±0.06ms 42.9 MB/sec
string_ree_95pct_null/default 1.00 24.7±0.06ms 56.9 MB/sec 1.00 24.8±0.05ms 56.7 MB/sec
string_ree_95pct_null/parquet_2 1.00 24.7±0.06ms 56.9 MB/sec 1.00 24.8±0.05ms 56.6 MB/sec
string_ree_95pct_null/zstd 1.00 25.6±0.07ms 55.1 MB/sec 1.00 25.7±0.06ms 54.8 MB/sec
string_ree_95pct_null/zstd_parquet_2 1.00 25.5±0.07ms 55.2 MB/sec 1.00 25.6±0.06ms 55.0 MB/sec
struct_all_null/bloom_filter 1.03 383.7±1.31µs 41.0 GB/sec 1.00 373.8±2.00µs 42.1 GB/sec
struct_all_null/cdc 1.00 7.7±0.06ms 2.0 GB/sec 1.03 7.9±0.19ms 2031.0 MB/sec
struct_all_null/default 1.00 116.3±0.51µs 135.4 GB/sec 1.00 116.7±0.74µs 135.0 GB/sec
struct_all_null/parquet_2 1.00 115.3±0.45µs 136.5 GB/sec 1.00 115.7±0.54µs 136.1 GB/sec
struct_all_null/zstd 1.01 168.6±0.54µs 93.4 GB/sec 1.00 167.5±0.64µs 94.0 GB/sec
struct_all_null/zstd_parquet_2 1.00 151.4±0.55µs 104.0 GB/sec 1.00 151.0±0.60µs 104.3 GB/sec
struct_non_null/bloom_filter 1.03 46.7±0.24ms 342.3 MB/sec 1.00 45.3±0.13ms 352.9 MB/sec
struct_non_null/cdc 1.01 45.1±0.15ms 354.5 MB/sec 1.00 44.8±0.19ms 356.8 MB/sec
struct_non_null/default 1.02 32.2±0.11ms 497.6 MB/sec 1.00 31.4±0.14ms 509.8 MB/sec
struct_non_null/parquet_2 1.01 40.6±0.11ms 394.0 MB/sec 1.00 40.1±0.12ms 398.6 MB/sec
struct_non_null/zstd 1.02 41.1±0.75ms 389.4 MB/sec 1.00 40.1±0.10ms 399.0 MB/sec
struct_non_null/zstd_parquet_2 1.01 54.6±0.11ms 293.2 MB/sec 1.00 54.2±0.11ms 295.4 MB/sec
struct_sparse_99pct_null/bloom_filter 1.00 6.5±0.02ms 2.4 GB/sec 1.04 6.8±0.02ms 2.3 GB/sec
struct_sparse_99pct_null/cdc 1.00 14.0±0.07ms 1150.1 MB/sec 1.01 14.2±0.10ms 1133.9 MB/sec
struct_sparse_99pct_null/default 1.00 5.9±0.01ms 2.6 GB/sec 1.05 6.3±0.01ms 2.5 GB/sec
struct_sparse_99pct_null/parquet_2 1.00 6.0±0.02ms 2.6 GB/sec 1.05 6.3±0.02ms 2.5 GB/sec
struct_sparse_99pct_null/zstd 1.00 7.3±0.02ms 2.2 GB/sec 1.04 7.6±0.02ms 2.1 GB/sec
struct_sparse_99pct_null/zstd_parquet_2 1.00 6.7±0.02ms 2.3 GB/sec 1.05 7.0±0.01ms 2.2 GB/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2970.7s |
| Peak memory | 2.9 GiB |
| Avg memory | 2.7 GiB |
| CPU user | 2915.7s |
| CPU sys | 51.6s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 2900.6s |
| Peak memory | 2.9 GiB |
| Avg memory | 2.6 GiB |
| CPU user | 2848.4s |
| CPU sys | 48.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3606592797
##########
parquet/benches/arrow_writer.rs:
##########
@@ -251,6 +251,22 @@ fn create_float_bench_batch_with_nans(size: usize, nan_density: f32) -> Result<R
)?)
}
+fn create_decimal_bench_batch(size: usize, null_density: f32) -> Result<RecordBatch> {
Review Comment:
Could you break the benchmarks out into a separate PR? 🙏 It would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5053492576
run benchmark arrow_writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5081001440
> There are a few other ideas worth exploring, primarily custom ColumnValueEncoders that converts on the fly to avoid the temporary allocs all together.
I suspect this would perform well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#issuecomment-5053508630
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10364#issuecomment-5053492576)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5053492576-1243-2qvj2 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing parquet-contiguous-decimal (0495502d2d1a821d383ad7112b7e108aa84457a9) to cd17899 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/cd17899fe4c68c812a9585ccaf4a36f1d30e0978..0495502d2d1a821d383ad7112b7e108aa84457a9)
BENCH_NAME=arrow_writer
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "MassivePizza (via GitHub)" <gi...@apache.org>.
MassivePizza commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3615872563
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1834,14 +1834,15 @@ fn get_interval_ym_array_slice(
array: &arrow_array::IntervalYearMonthArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
+ const VALUE_SIZE: usize = 12;
+ let mut arena = Vec::with_capacity(indices.len() * VALUE_SIZE);
for i in indices {
- let mut value = array.value(i).to_le_bytes().to_vec();
- let mut suffix = vec![0; 8];
- value.append(&mut suffix);
- values.push(FixedLenByteArray::from(ByteArray::from(value)))
+ let mut out = [0; 12];
+ let value = array.value(i);
+ out[0..4].copy_from_slice(&value.to_le_bytes());
+ arena.extend_from_slice(&out);
Review Comment:
Looked at the codegen, and it avoided some XORs. The real benefit came from rewriting it to use `chunks_exact_mut` (see https://github.com/apache/arrow-rs/pull/10364/changes/baa13dd29c458595eb6913fc2a24885309cbf6c1).
And I preferred the per-value XOR above a zeroed alloc, since the chunk is being filled linearly anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3651204835
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1837,93 +1833,111 @@ fn get_interval_dt_array_slice(
array: &arrow_array::IntervalDayTimeArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
- for i in indices {
- let mut out = [0; 12];
+ chunk_array_slice(12, indices, move |i, chunk| {
let value = array.value(i);
- out[4..8].copy_from_slice(&value.days.to_le_bytes());
- out[8..12].copy_from_slice(&value.milliseconds.to_le_bytes());
- values.push(FixedLenByteArray::from(ByteArray::from(out.to_vec())));
+ chunk[4..8].copy_from_slice(&value.days.to_le_bytes());
+ chunk[8..12].copy_from_slice(&value.milliseconds.to_le_bytes());
+ })
+}
+
+trait NativeDecimalType: DecimalType {
Review Comment:
Maybe we could just add these to the DecimalType trait itself (rather than a private trait) -- perhaps as a follow on PR
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1821,14 +1821,10 @@ fn get_interval_ym_array_slice(
array: &arrow_array::IntervalYearMonthArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
- for i in indices {
- let mut value = array.value(i).to_le_bytes().to_vec();
- let mut suffix = vec![0; 8];
Review Comment:
that is indeed an expensive way to append 8 bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "MassivePizza (via GitHub)" <gi...@apache.org>.
MassivePizza commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3614351024
##########
parquet/src/column/writer/mod.rs:
##########
@@ -1649,12 +1649,16 @@ fn update_max<T: ParquetValueType>(descr: &ColumnDescriptor, val: &T, max: &mut
update_stat::<T, _>(descr, val, max, |cur| compare_greater(descr, val, cur))
}
-#[inline]
-#[allow(clippy::eq_op)]
fn is_nan<T: ParquetValueType>(descr: &ColumnDescriptor, val: &T) -> bool {
+ is_nan_with(descr.logical_type_ref(), val)
+}
+
+#[inline(always)]
+#[allow(clippy::eq_op)]
+fn is_nan_with<T: ParquetValueType>(logical_type_ref: Option<&LogicalType>, val: &T) -> bool {
Review Comment:
Fair enough, will split it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): slice up contiguous buffer for decimals and fsb [arrow-rs]
Posted by "MassivePizza (via GitHub)" <gi...@apache.org>.
MassivePizza commented on code in PR #10364:
URL: https://github.com/apache/arrow-rs/pull/10364#discussion_r3615872563
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -1834,14 +1834,15 @@ fn get_interval_ym_array_slice(
array: &arrow_array::IntervalYearMonthArray,
indices: impl ExactSizeIterator<Item = usize>,
) -> Vec<FixedLenByteArray> {
- let mut values = Vec::with_capacity(indices.len());
+ const VALUE_SIZE: usize = 12;
+ let mut arena = Vec::with_capacity(indices.len() * VALUE_SIZE);
for i in indices {
- let mut value = array.value(i).to_le_bytes().to_vec();
- let mut suffix = vec![0; 8];
- value.append(&mut suffix);
- values.push(FixedLenByteArray::from(ByteArray::from(value)))
+ let mut out = [0; 12];
+ let value = array.value(i);
+ out[0..4].copy_from_slice(&value.to_le_bytes());
+ arena.extend_from_slice(&out);
Review Comment:
Looked at the codegen, and it avoided some XORs. The real benefit came from rewriting it to use `chunks_exact_mut.` (see https://github.com/apache/arrow-rs/pull/10364/changes/baa13dd29c458595eb6913fc2a24885309cbf6c1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org