You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/07/12 12:18:29 UTC
[PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Dandandan opened a new pull request, #23492:
URL: https://github.com/apache/datafusion/pull/23492
## Which issue does this PR close?
- None. This is an experimental performance enhancement with benchmark results included for review.
## Rationale for this change
Parquet scans currently request the projected column chunks for each row group as the push decoder reaches it. For object stores with meaningful per-request latency, those sequential I/O requests can dominate query time even when the total amount of data is small.
This change allows scans to batch I/O for complete subsequent row groups while keeping decoding lazy and enforcing a configurable byte budget.
## What changes are included in this PR?
- Add `datafusion.execution.parquet.prefetch_size`, an optional byte budget that is disabled by default.
- Use the push decoder's `buffered_bytes()` accounting when deciding how much speculative data can be staged.
- Always fetch the ranges required by the current row group, even when they exceed the budget.
- Append projected column ranges for complete subsequent row groups in scan order while `buffered + required + speculative` bytes fit the budget.
- Keep prefetching I/O-only: prefetched row groups remain encoded in the push decoder until normal execution requests them.
- Propagate the option through Parquet configuration and protobuf serialization.
- Document the option and include the complete TPC-H and ClickBench per-query benchmark report in `benchmarks/parquet_prefetch_benchmark.md`.
### Benchmark results
The benchmarks use three iterations, TPC-H SF1, a 167 MiB single-file ClickBench dataset, and the existing simulated S3-like latency wrapper (25-200 ms per GET). Speedup is prefetch-off time divided by 20 MiB prefetch time.
| Configuration | TPC-H workload speedup | ClickBench workload speedup |
|---|---:|---:|
| Local I/O, filter pushdown off | 0.99x | 1.03x |
| Simulated latency, filter pushdown off | 1.94x | 1.21x |
| Local I/O, filter pushdown + reorder on | 1.02x | 1.01x |
| Simulated latency, filter pushdown + reorder on | 1.73x | 1.18x |
Increasing the latency-enabled budget from 20 MiB to 100 MiB did not improve the aggregate result:
| Benchmark | Off | 20 MiB | 100 MiB |
|---|---:|---:|---:|
| TPC-H SF1 | 53082.86 ms | 30651.08 ms | 30688.45 ms |
| ClickBench | 18852.91 ms | 15916.91 ms | 16100.72 ms |
The full report contains setup details and every per-query timing.
## Are these changes tested?
Yes.
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
- The prescribed extended workspace test command compiled and ran the Rust unit, integration, fuzz, and Parquet suites successfully. It identified the expected `df_settings` snapshot change introduced by the new option.
- After updating that snapshot, the complete 493-file SQL logic suite passed.
- Added focused push-decoder tests for budget packing, already-buffered byte accounting, disabled prefetch behavior, and staging prefetched row groups without another I/O request.
- Benchmark query result row counts matched between prefetch-off and prefetch-on runs.
## Are there any user-facing changes?
Yes. Users can opt in by setting `datafusion.execution.parquet.prefetch_size` to a byte budget such as `20971520` for 20 MiB. The default remains disabled, so existing behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951252059
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpch_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 38.07 / 38.90 ยฑ1.11 / 41.04 ms โ 37.84 / 38.73 ยฑ0.91 / 40.41 ms โ no change โ
โ QQuery 2 โ 18.88 / 19.64 ยฑ1.14 / 21.89 ms โ 18.84 / 19.24 ยฑ0.31 / 19.76 ms โ no change โ
โ QQuery 3 โ 30.61 / 32.58 ยฑ1.94 / 35.88 ms โ 30.61 / 33.54 ยฑ1.80 / 36.28 ms โ no change โ
โ QQuery 4 โ 17.37 / 17.57 ยฑ0.25 / 18.04 ms โ 17.30 / 17.52 ยฑ0.27 / 18.03 ms โ no change โ
โ QQuery 5 โ 37.70 / 40.36 ยฑ2.27 / 43.10 ms โ 39.85 / 40.57 ยฑ0.40 / 40.92 ms โ no change โ
โ QQuery 6 โ 16.05 / 16.68 ยฑ0.89 / 18.42 ms โ 15.94 / 16.32 ยฑ0.30 / 16.81 ms โ no change โ
โ QQuery 7 โ 44.11 / 45.40 ยฑ1.67 / 48.55 ms โ 44.12 / 45.84 ยฑ1.62 / 48.64 ms โ no change โ
โ QQuery 8 โ 42.68 / 43.24 ยฑ0.92 / 45.06 ms โ 42.55 / 43.07 ยฑ0.37 / 43.62 ms โ no change โ
โ QQuery 9 โ 49.15 / 50.66 ยฑ2.18 / 54.94 ms โ 48.61 / 49.32 ยฑ0.52 / 50.17 ms โ no change โ
โ QQuery 10 โ 41.84 / 41.99 ยฑ0.13 / 42.23 ms โ 42.01 / 42.69 ยฑ1.05 / 44.79 ms โ no change โ
โ QQuery 11 โ 13.37 / 13.96 ยฑ0.57 / 14.75 ms โ 13.26 / 13.74 ยฑ0.65 / 15.03 ms โ no change โ
โ QQuery 12 โ 23.52 / 24.41 ยฑ0.71 / 25.37 ms โ 23.74 / 24.99 ยฑ1.14 / 26.45 ms โ no change โ
โ QQuery 13 โ 32.11 / 33.97 ยฑ2.11 / 37.84 ms โ 32.78 / 34.71 ยฑ1.30 / 36.17 ms โ no change โ
โ QQuery 14 โ 23.56 / 23.82 ยฑ0.14 / 23.94 ms โ 23.43 / 23.73 ยฑ0.25 / 24.18 ms โ no change โ
โ QQuery 15 โ 30.81 / 31.64 ยฑ0.93 / 32.93 ms โ 31.24 / 31.41 ยฑ0.15 / 31.61 ms โ no change โ
โ QQuery 16 โ 13.96 / 14.34 ยฑ0.52 / 15.35 ms โ 13.74 / 13.93 ยฑ0.19 / 14.21 ms โ no change โ
โ QQuery 17 โ 73.23 / 75.93 ยฑ1.78 / 78.34 ms โ 72.62 / 73.91 ยฑ1.44 / 75.93 ms โ no change โ
โ QQuery 18 โ 58.62 / 60.79 ยฑ1.35 / 62.88 ms โ 57.47 / 60.60 ยฑ3.06 / 65.34 ms โ no change โ
โ QQuery 19 โ 32.73 / 34.03 ยฑ1.28 / 36.03 ms โ 32.79 / 33.83 ยฑ1.18 / 35.77 ms โ no change โ
โ QQuery 20 โ 31.68 / 32.22 ยฑ0.45 / 32.93 ms โ 31.79 / 32.14 ยฑ0.19 / 32.38 ms โ no change โ
โ QQuery 21 โ 54.29 / 55.11 ยฑ0.79 / 56.15 ms โ 54.40 / 56.11 ยฑ1.14 / 57.66 ms โ no change โ
โ QQuery 22 โ 13.55 / 13.84 ยฑ0.22 / 14.21 ms โ 13.72 / 14.06 ยฑ0.44 / 14.91 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 761.09ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 760.01ms โ
โ Average Time (HEAD) โ 34.59ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 34.55ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 22 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 516.3 MiB |
| CPU user | 21.9s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
**tpch โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 520.7 MiB |
| CPU user | 22.0s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951815995
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpch_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 573.50 / 616.27 ยฑ35.03 / 662.98 ms โ 212.90 / 225.87 ยฑ10.62 / 238.27 ms โ +2.73x faster โ
โ QQuery 2 โ 787.96 / 875.87 ยฑ49.01 / 935.71 ms โ 692.85 / 747.96 ยฑ43.03 / 799.29 ms โ +1.17x faster โ
โ QQuery 3 โ 947.45 / 1011.14 ยฑ46.33 / 1074.99 ms โ 511.42 / 558.97 ยฑ36.95 / 609.07 ms โ +1.81x faster โ
โ QQuery 4 โ 734.55 / 804.89 ยฑ66.27 / 902.88 ms โ 397.37 / 401.78 ยฑ7.89 / 417.55 ms โ +2.00x faster โ
โ QQuery 5 โ 1074.50 / 1210.11 ยฑ109.24 / 1373.13 ms โ 703.02 / 775.31 ยฑ41.85 / 824.53 ms โ +1.56x faster โ
โ QQuery 6 โ 531.42 / 575.99 ยฑ39.13 / 631.95 ms โ 194.29 / 208.36 ยฑ10.83 / 221.72 ms โ +2.76x faster โ
โ QQuery 7 โ 981.75 / 1136.33 ยฑ105.76 / 1247.37 ms โ 604.58 / 772.00 ยฑ97.78 / 886.68 ms โ +1.47x faster โ
โ QQuery 8 โ 1121.66 / 1280.33 ยฑ136.01 / 1458.29 ms โ 833.47 / 874.11 ยฑ21.27 / 895.77 ms โ +1.46x faster โ
โ QQuery 9 โ 1126.28 / 1209.32 ยฑ51.09 / 1278.37 ms โ 839.29 / 841.14 ยฑ1.16 / 842.49 ms โ +1.44x faster โ
โ QQuery 10 โ 913.75 / 1014.22 ยฑ54.61 / 1060.92 ms โ 526.77 / 551.86 ยฑ24.97 / 589.36 ms โ +1.84x faster โ
โ QQuery 11 โ 669.93 / 818.52 ยฑ100.21 / 968.57 ms โ 615.77 / 691.70 ยฑ44.58 / 754.40 ms โ +1.18x faster โ
โ QQuery 12 โ 746.30 / 840.99 ยฑ60.69 / 909.27 ms โ 403.69 / 408.94 ยฑ6.21 / 420.56 ms โ +2.06x faster โ
โ QQuery 13 โ 399.96 / 427.32 ยฑ30.87 / 465.21 ms โ 320.65 / 361.38 ยฑ36.00 / 402.71 ms โ +1.18x faster โ
โ QQuery 14 โ 662.16 / 704.04 ยฑ45.71 / 787.09 ms โ 236.25 / 327.04 ยฑ57.34 / 409.68 ms โ +2.15x faster โ
โ QQuery 15 โ 1165.64 / 1228.56 ยฑ58.47 / 1304.60 ms โ 443.52 / 540.92 ยฑ78.98 / 613.88 ms โ +2.27x faster โ
โ QQuery 16 โ 377.21 / 413.33 ยฑ37.05 / 483.09 ms โ 264.62 / 326.51 ยฑ57.74 / 394.73 ms โ +1.27x faster โ
โ QQuery 17 โ 1098.42 / 1309.29 ยฑ147.52 / 1519.69 ms โ 515.28 / 563.14 ยฑ35.57 / 623.78 ms โ +2.32x faster โ
โ QQuery 18 โ 1379.50 / 1546.22 ยฑ120.75 / 1718.06 ms โ 801.71 / 809.93 ยฑ5.35 / 818.15 ms โ +1.91x faster โ
โ QQuery 19 โ 572.62 / 680.62 ยฑ84.83 / 822.46 ms โ 278.85 / 345.19 ยฑ42.94 / 414.57 ms โ +1.97x faster โ
โ QQuery 20 โ 883.88 / 1027.48 ยฑ76.20 / 1103.98 ms โ 539.15 / 576.08 ยฑ34.98 / 636.89 ms โ +1.78x faster โ
โ QQuery 21 โ 2049.11 / 2185.75 ยฑ81.40 / 2279.74 ms โ 945.61 / 952.67 ยฑ4.70 / 959.98 ms โ +2.29x faster โ
โ QQuery 22 โ 496.08 / 536.31 ยฑ31.22 / 584.09 ms โ 494.79 / 497.12 ยฑ1.55 / 499.04 ms โ +1.08x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 21452.91ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 12357.99ms โ
โ Average Time (HEAD) โ 975.13ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 561.73ms โ
โ Queries Faster โ 22 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 0 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 801.3 MiB |
| Avg memory | 402.1 MiB |
| CPU user | 29.8s |
| CPU sys | 3.8s |
| Peak spill | 0 B |
**tpch โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 65.0s |
| Peak memory | 892.1 MiB |
| Avg memory | 450.9 MiB |
| CPU user | 29.4s |
| CPU sys | 3.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951505403
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 106.58 / 173.19 ยฑ37.74 / 207.13 ms โ 106.62 / 173.05 ยฑ37.83 / 207.06 ms โ no change โ
โ QQuery 2 โ 520.99 / 648.79 ยฑ69.02 / 707.70 ms โ 522.66 / 656.64 ยฑ71.45 / 707.45 ms โ no change โ
โ QQuery 3 โ 381.19 / 426.92 ยฑ31.95 / 473.61 ms โ 382.84 / 429.45 ยฑ30.32 / 473.13 ms โ no change โ
โ QQuery 4 โ 1201.76 / 1268.72 ยฑ55.15 / 1345.91 ms โ 1184.88 / 1280.51 ยฑ69.32 / 1355.09 ms โ no change โ
โ QQuery 5 โ 516.56 / 583.71 ยฑ53.80 / 677.18 ms โ 554.15 / 595.17 ยฑ49.79 / 689.32 ms โ no change โ
โ QQuery 6 โ 545.98 / 696.44 ยฑ102.40 / 822.34 ms โ 541.11 / 700.27 ยฑ105.90 / 817.30 ms โ no change โ
โ QQuery 7 โ 566.53 / 692.09 ยฑ70.16 / 759.43 ms โ 547.07 / 663.07 ยฑ72.36 / 772.69 ms โ no change โ
โ QQuery 8 โ 572.51 / 608.18 ยฑ31.15 / 652.73 ms โ 483.31 / 569.71 ยฑ57.82 / 642.67 ms โ +1.07x faster โ
โ QQuery 9 โ 1372.25 / 1491.12 ยฑ96.71 / 1592.41 ms โ 1378.59 / 1491.36 ยฑ93.22 / 1589.76 ms โ no change โ
โ QQuery 10 โ 728.49 / 756.17 ยฑ21.51 / 785.52 ms โ 751.64 / 783.04 ยฑ31.77 / 841.55 ms โ no change โ
โ QQuery 11 โ 928.13 / 1030.56 ยฑ74.92 / 1136.87 ms โ 930.96 / 1007.77 ยฑ44.29 / 1051.23 ms โ no change โ
โ QQuery 12 โ 346.62 / 374.99 ยฑ26.30 / 418.19 ms โ 343.62 / 372.35 ยฑ27.19 / 418.87 ms โ no change โ
โ QQuery 13 โ 661.82 / 742.25 ยฑ47.83 / 796.12 ms โ 678.82 / 749.29 ยฑ42.15 / 789.21 ms โ no change โ
โ QQuery 14 โ 4051.16 / 4130.38 ยฑ40.94 / 4166.10 ms โ 3750.80 / 3998.85 ยฑ191.80 / 4285.01 ms โ no change โ
โ QQuery 15 โ 407.43 / 498.97 ยฑ49.31 / 547.76 ms โ 405.36 / 498.39 ยฑ49.92 / 548.52 ms โ no change โ
โ QQuery 16 โ 108.07 / 173.96 ยฑ37.55 / 208.04 ms โ 108.08 / 174.86 ยฑ38.29 / 211.34 ms โ no change โ
โ QQuery 17 โ 618.80 / 708.98 ยฑ48.71 / 758.61 ms โ 636.25 / 709.18 ยฑ41.46 / 756.50 ms โ no change โ
โ QQuery 18 โ 716.99 / 800.69 ยฑ60.78 / 903.69 ms โ 718.64 / 804.29 ยฑ61.47 / 908.25 ms โ no change โ
โ QQuery 19 โ 533.49 / 599.91 ยฑ56.47 / 676.26 ms โ 485.21 / 581.13 ยฑ60.99 / 648.99 ms โ no change โ
โ QQuery 20 โ 326.95 / 400.03 ยฑ41.53 / 452.35 ms โ 325.16 / 398.38 ยฑ40.75 / 447.53 ms โ no change โ
โ QQuery 21 โ 507.32 / 572.91 ยฑ49.77 / 634.03 ms โ 508.42 / 571.64 ยฑ47.90 / 628.08 ms โ no change โ
โ QQuery 22 โ 473.16 / 526.56 ยฑ32.61 / 573.88 ms โ 477.99 / 527.70 ยฑ30.10 / 570.34 ms โ no change โ
โ QQuery 23 โ 2546.90 / 2680.19 ยฑ87.28 / 2759.78 ms โ 2467.53 / 2689.09 ยฑ129.37 / 2827.59 ms โ no change โ
โ QQuery 24 โ 2178.14 / 2353.72 ยฑ107.81 / 2470.57 ms โ 2192.59 / 2357.34 ยฑ104.69 / 2477.86 ms โ no change โ
โ QQuery 25 โ 683.41 / 731.96 ยฑ35.37 / 770.57 ms โ 687.55 / 732.63 ยฑ37.10 / 777.97 ms โ no change โ
โ QQuery 26 โ 554.11 / 612.15 ยฑ31.67 / 638.26 ms โ 559.93 / 621.93 ยฑ48.89 / 701.66 ms โ no change โ
โ QQuery 27 โ 145.32 / 254.81 ยฑ82.74 / 359.16 ms โ 141.28 / 254.27 ยฑ84.30 / 359.61 ms โ no change โ
โ QQuery 28 โ 340.14 / 404.43 ยฑ35.21 / 440.08 ms โ 344.10 / 401.00 ยฑ31.88 / 433.98 ms โ no change โ
โ QQuery 29 โ 721.09 / 785.39 ยฑ42.21 / 823.69 ms โ 720.83 / 788.90 ยฑ49.25 / 845.60 ms โ no change โ
โ QQuery 30 โ 498.15 / 633.53 ยฑ83.59 / 745.07 ms โ 550.99 / 632.52 ยฑ74.36 / 743.39 ms โ no change โ
โ QQuery 31 โ 1217.23 / 1316.93 ยฑ63.89 / 1411.04 ms โ 1209.55 / 1303.29 ยฑ48.47 / 1350.40 ms โ no change โ
โ QQuery 32 โ 414.37 / 443.80 ยฑ25.90 / 484.75 ms โ 413.47 / 443.27 ยฑ26.32 / 485.71 ms โ no change โ
โ QQuery 33 โ 520.67 / 594.34 ยฑ43.53 / 644.76 ms โ 522.71 / 594.73 ยฑ41.87 / 640.84 ms โ no change โ
โ QQuery 34 โ 194.44 / 306.30 ยฑ59.75 / 358.52 ms โ 193.27 / 305.72 ยฑ60.22 / 359.48 ms โ no change โ
โ QQuery 35 โ 713.37 / 751.72 ยฑ32.52 / 801.63 ms โ 652.37 / 757.92 ยฑ71.76 / 874.26 ms โ no change โ
โ QQuery 36 โ 107.47 / 173.96 ยฑ38.15 / 210.44 ms โ 107.29 / 174.01 ยฑ38.03 / 207.89 ms โ no change โ
โ QQuery 37 โ 221.50 / 276.50 ยฑ30.10 / 310.05 ms โ 221.14 / 276.54 ยฑ30.37 / 310.38 ms โ no change โ
โ QQuery 38 โ 522.74 / 653.52 ยฑ69.70 / 725.65 ms โ 661.80 / 714.27 ยฑ52.79 / 813.39 ms โ 1.09x slower โ
โ QQuery 39 โ 1851.53 / 2010.97 ยฑ112.63 / 2154.59 ms โ 1853.46 / 2011.00 ยฑ111.72 / 2147.70 ms โ no change โ
โ QQuery 40 โ 466.61 / 493.06 ยฑ20.59 / 510.48 ms โ 466.43 / 493.10 ยฑ21.19 / 510.99 ms โ no change โ
โ QQuery 41 โ 176.03 / 278.26 ยฑ73.02 / 380.60 ms โ 176.22 / 279.11 ยฑ73.17 / 381.42 ms โ no change โ
โ QQuery 42 โ 314.71 / 406.43 ยฑ65.22 / 474.04 ms โ 313.84 / 406.07 ยฑ65.70 / 468.16 ms โ no change โ
โ QQuery 43 โ 106.56 / 153.47 ยฑ40.37 / 207.44 ms โ 106.76 / 152.76 ยฑ40.64 / 206.50 ms โ no change โ
โ QQuery 44 โ 197.33 / 262.38 ยฑ47.60 / 302.46 ms โ 197.54 / 262.85 ยฑ47.52 / 303.47 ms โ no change โ
โ QQuery 45 โ 487.64 / 567.00 ยฑ51.27 / 644.98 ms โ 452.22 / 541.16 ยฑ67.39 / 647.78 ms โ no change โ
โ QQuery 46 โ 309.53 / 351.40 ยฑ34.83 / 399.07 ms โ 290.02 / 345.95 ยฑ41.70 / 397.85 ms โ no change โ
โ QQuery 47 โ 1233.32 / 1273.36 ยฑ41.79 / 1331.29 ms โ 1243.07 / 1277.95 ยฑ34.25 / 1325.17 ms โ no change โ
โ QQuery 48 โ 589.33 / 703.40 ยฑ63.71 / 767.41 ms โ 596.08 / 705.31 ยฑ65.07 / 774.86 ms โ no change โ
โ QQuery 49 โ 581.69 / 634.95 ยฑ48.32 / 702.23 ms โ 585.40 / 647.06 ยฑ45.19 / 706.56 ms โ no change โ
โ QQuery 50 โ 541.38 / 633.13 ยฑ49.73 / 678.52 ms โ 536.47 / 638.78 ยฑ54.24 / 691.36 ms โ no change โ
โ QQuery 51 โ 512.78 / 539.91 ยฑ18.29 / 557.91 ms โ 503.40 / 539.53 ยฑ28.83 / 565.73 ms โ no change โ
โ QQuery 52 โ 313.13 / 406.88 ยฑ67.05 / 475.32 ms โ 308.24 / 402.52 ยฑ65.90 / 469.19 ms โ no change โ
โ QQuery 53 โ 412.65 / 453.24 ยฑ47.30 / 515.52 ms โ 413.16 / 453.60 ยฑ48.61 / 513.85 ms โ no change โ
โ QQuery 54 โ 903.69 / 938.74 ยฑ33.52 / 996.88 ms โ 919.95 / 978.05 ยฑ40.19 / 1036.27 ms โ no change โ
โ QQuery 55 โ 377.89 / 424.99 ยฑ30.57 / 466.72 ms โ 376.16 / 424.66 ยฑ31.53 / 469.85 ms โ no change โ
โ QQuery 56 โ 539.08 / 614.93 ยฑ52.77 / 691.87 ms โ 523.17 / 584.63 ยฑ35.25 / 623.56 ms โ no change โ
โ QQuery 57 โ 1111.73 / 1193.23 ยฑ60.25 / 1276.10 ms โ 1083.06 / 1190.79 ยฑ80.41 / 1308.30 ms โ no change โ
โ QQuery 58 โ 1155.87 / 1326.89 ยฑ96.27 / 1423.05 ms โ 1154.35 / 1327.87 ยฑ97.50 / 1425.56 ms โ no change โ
โ QQuery 59 โ 907.17 / 928.63 ยฑ41.79 / 1012.20 ms โ 906.65 / 932.97 ยฑ48.49 / 1029.84 ms โ no change โ
โ QQuery 60 โ 580.21 / 669.50 ยฑ60.32 / 742.09 ms โ 575.99 / 657.50 ยฑ54.62 / 740.92 ms โ no change โ
โ QQuery 61 โ 393.54 / 394.93 ยฑ0.73 / 395.63 ms โ 394.41 / 395.06 ยฑ0.48 / 395.58 ms โ no change โ
โ QQuery 62 โ 580.07 / 600.53 ยฑ24.64 / 631.75 ms โ 579.20 / 600.48 ยฑ25.13 / 631.92 ms โ no change โ
โ QQuery 63 โ 416.63 / 440.79 ยฑ28.49 / 475.76 ms โ 417.81 / 441.41 ยฑ27.97 / 478.06 ms โ no change โ
โ QQuery 64 โ 3163.42 / 3300.85 ยฑ92.90 / 3408.81 ms โ 3185.37 / 3291.29 ยฑ74.53 / 3385.49 ms โ no change โ
โ QQuery 65 โ 639.73 / 741.84 ยฑ60.79 / 813.56 ms โ 626.46 / 739.25 ยฑ66.74 / 828.17 ms โ no change โ
โ QQuery 66 โ 562.08 / 592.08 ยฑ46.81 / 685.41 ms โ 562.65 / 595.68 ยฑ41.36 / 675.49 ms โ no change โ
โ QQuery 67 โ 638.53 / 703.83 ยฑ44.33 / 751.49 ms โ 618.40 / 686.97 ยฑ45.85 / 737.67 ms โ no change โ
โ QQuery 68 โ 283.56 / 317.83 ยฑ19.44 / 338.42 ms โ 284.24 / 313.24 ยฑ15.64 / 326.57 ms โ no change โ
โ QQuery 69 โ 624.68 / 774.30 ยฑ109.53 / 931.83 ms โ 697.82 / 779.06 ยฑ53.48 / 862.06 ms โ no change โ
โ QQuery 70 โ 684.00 / 759.38 ยฑ61.29 / 856.30 ms โ 694.83 / 763.45 ยฑ56.21 / 855.44 ms โ no change โ
โ QQuery 71 โ 486.71 / 564.89 ยฑ62.46 / 668.23 ms โ 514.52 / 591.98 ยฑ70.70 / 676.42 ms โ no change โ
โ QQuery 72 โ 2916.19 / 3106.91 ยฑ104.87 / 3207.29 ms โ 2944.67 / 3067.50 ยฑ77.33 / 3173.44 ms โ no change โ
โ QQuery 73 โ 194.76 / 308.62 ยฑ58.89 / 359.19 ms โ 194.33 / 307.21 ยฑ60.82 / 357.65 ms โ no change โ
โ QQuery 74 โ 681.88 / 800.19 ยฑ85.43 / 936.92 ms โ 741.89 / 810.40 ยฑ59.17 / 918.42 ms โ no change โ
โ QQuery 75 โ 941.40 / 982.65 ยฑ31.65 / 1039.29 ms โ 939.80 / 997.15 ยฑ40.34 / 1045.65 ms โ no change โ
โ QQuery 76 โ 515.11 / 598.93 ยฑ56.05 / 657.67 ms โ 516.97 / 598.22 ยฑ56.89 / 655.35 ms โ no change โ
โ QQuery 77 โ 643.49 / 718.76 ยฑ59.96 / 813.47 ms โ 685.82 / 747.77 ยฑ44.19 / 797.45 ms โ no change โ
โ QQuery 78 โ 709.42 / 859.81 ยฑ82.97 / 961.14 ms โ 706.01 / 855.88 ยฑ81.92 / 954.60 ms โ no change โ
โ QQuery 79 โ 451.54 / 500.08 ยฑ45.23 / 582.67 ms โ 452.44 / 505.92 ยฑ50.57 / 599.34 ms โ no change โ
โ QQuery 80 โ 638.23 / 710.80 ยฑ64.69 / 794.09 ms โ 640.47 / 716.23 ยฑ50.13 / 794.67 ms โ no change โ
โ QQuery 81 โ 488.87 / 604.10 ยฑ70.91 / 685.21 ms โ 477.80 / 638.71 ยฑ81.07 / 690.62 ms โ 1.06x slower โ
โ QQuery 82 โ 557.08 / 643.60 ยฑ47.53 / 682.13 ms โ 557.41 / 644.79 ยฑ47.92 / 682.65 ms โ no change โ
โ QQuery 83 โ 982.04 / 1015.36 ยฑ45.48 / 1098.42 ms โ 860.55 / 1013.13 ยฑ87.71 / 1097.65 ms โ no change โ
โ QQuery 84 โ 440.52 / 503.36 ยฑ57.85 / 595.64 ms โ 432.77 / 501.41 ยฑ59.59 / 595.80 ms โ no change โ
โ QQuery 85 โ 822.30 / 891.55 ยฑ50.67 / 951.27 ms โ 814.27 / 886.32 ยฑ51.04 / 951.85 ms โ no change โ
โ QQuery 86 โ 252.76 / 315.20 ยฑ52.68 / 390.24 ms โ 246.79 / 314.23 ยฑ55.37 / 391.06 ms โ no change โ
โ QQuery 87 โ 659.90 / 682.30 ยฑ30.63 / 740.93 ms โ 568.16 / 679.97 ยฑ84.74 / 780.02 ms โ no change โ
โ QQuery 88 โ 737.47 / 739.69 ยฑ2.32 / 743.70 ms โ 738.41 / 739.84 ยฑ1.36 / 741.94 ms โ no change โ
โ QQuery 89 โ 425.29 / 460.88 ยฑ28.19 / 485.29 ms โ 426.70 / 459.86 ยฑ27.04 / 482.95 ms โ no change โ
โ QQuery 90 โ 426.72 / 466.89 ยฑ26.55 / 504.33 ms โ 425.27 / 466.40 ยฑ25.76 / 499.66 ms โ no change โ
โ QQuery 91 โ 523.47 / 663.29 ยฑ112.45 / 807.77 ms โ 526.00 / 664.16 ยฑ112.23 / 806.66 ms โ no change โ
โ QQuery 92 โ 372.92 / 425.33 ยฑ44.86 / 487.90 ms โ 371.37 / 418.19 ยฑ55.20 / 486.88 ms โ no change โ
โ QQuery 93 โ 465.94 / 481.16 ยฑ15.90 / 509.12 ms โ 470.08 / 482.52 ยฑ13.05 / 504.31 ms โ no change โ
โ QQuery 94 โ 481.27 / 528.77 ยฑ41.59 / 585.26 ms โ 479.38 / 529.30 ยฑ43.87 / 587.84 ms โ no change โ
โ QQuery 95 โ 691.53 / 776.13 ยฑ46.19 / 825.17 ms โ 694.67 / 778.67 ยฑ45.34 / 823.84 ms โ no change โ
โ QQuery 96 โ 407.64 / 446.64 ยฑ47.11 / 506.77 ms โ 407.93 / 447.25 ยฑ47.57 / 507.98 ms โ no change โ
โ QQuery 97 โ 459.05 / 519.48 ยฑ49.68 / 610.74 ms โ 454.41 / 518.76 ยฑ50.53 / 610.06 ms โ no change โ
โ QQuery 98 โ 395.69 / 440.96 ยฑ32.33 / 486.50 ms โ 395.72 / 441.72 ยฑ29.93 / 485.33 ms โ no change โ
โ QQuery 99 โ 508.23 / 611.98 ยฑ56.66 / 658.90 ms โ 508.99 / 620.09 ยฑ63.68 / 695.76 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 75138.17ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 75086.23ms โ
โ Average Time (HEAD) โ 758.97ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 758.45ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 96 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 1.0 GiB |
| Avg memory | 513.9 MiB |
| CPU user | 239.0s |
| CPU sys | 12.0s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 1.1 GiB |
| Avg memory | 578.0 MiB |
| CPU user | 236.4s |
| CPU sys | 11.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951260777
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 5.75 / 6.22 ยฑ0.80 / 7.81 ms โ 5.62 / 6.14 ยฑ0.84 / 7.82 ms โ no change โ
โ QQuery 2 โ 81.85 / 82.63 ยฑ0.50 / 83.33 ms โ 82.37 / 82.53 ยฑ0.13 / 82.67 ms โ no change โ
โ QQuery 3 โ 29.89 / 30.51 ยฑ0.36 / 30.96 ms โ 30.42 / 30.99 ยฑ0.58 / 32.06 ms โ no change โ
โ QQuery 4 โ 548.66 / 558.16 ยฑ7.47 / 569.33 ms โ 510.58 / 520.02 ยฑ10.42 / 539.68 ms โ +1.07x faster โ
โ QQuery 5 โ 53.62 / 54.60 ยฑ1.26 / 57.09 ms โ 52.72 / 53.10 ยฑ0.39 / 53.84 ms โ no change โ
โ QQuery 6 โ 37.37 / 38.28 ยฑ0.57 / 38.89 ms โ 37.91 / 38.43 ยฑ0.38 / 38.88 ms โ no change โ
โ QQuery 7 โ 96.05 / 98.36 ยฑ1.57 / 100.99 ms โ 96.20 / 96.93 ยฑ0.80 / 98.47 ms โ no change โ
โ QQuery 8 โ 37.51 / 38.22 ยฑ0.53 / 38.97 ms โ 37.32 / 38.24 ยฑ0.73 / 39.21 ms โ no change โ
โ QQuery 9 โ 54.93 / 55.96 ยฑ0.68 / 57.03 ms โ 54.24 / 55.70 ยฑ1.11 / 57.03 ms โ no change โ
โ QQuery 10 โ 64.53 / 66.48 ยฑ2.49 / 71.33 ms โ 64.20 / 64.69 ยฑ0.30 / 64.96 ms โ no change โ
โ QQuery 11 โ 350.11 / 356.32 ยฑ4.09 / 361.41 ms โ 329.08 / 332.64 ยฑ2.68 / 337.06 ms โ +1.07x faster โ
โ QQuery 12 โ 30.84 / 32.84 ยฑ2.81 / 38.31 ms โ 29.58 / 29.82 ยฑ0.21 / 30.16 ms โ +1.10x faster โ
โ QQuery 13 โ 121.93 / 123.89 ยฑ1.75 / 126.73 ms โ 120.08 / 123.28 ยฑ4.17 / 131.53 ms โ no change โ
โ QQuery 14 โ 418.95 / 422.89 ยฑ4.55 / 431.32 ms โ 418.50 / 424.20 ยฑ4.79 / 431.31 ms โ no change โ
โ QQuery 15 โ 64.69 / 69.21 ยฑ4.96 / 78.87 ms โ 61.21 / 63.25 ยฑ1.27 / 64.80 ms โ +1.09x faster โ
โ QQuery 16 โ 7.08 / 7.17 ยฑ0.06 / 7.22 ms โ 6.77 / 6.95 ยฑ0.13 / 7.16 ms โ no change โ
โ QQuery 17 โ 82.18 / 83.88 ยฑ2.30 / 88.42 ms โ 82.31 / 83.74 ยฑ1.30 / 86.01 ms โ no change โ
โ QQuery 18 โ 124.87 / 127.27 ยฑ2.04 / 130.82 ms โ 126.24 / 127.87 ยฑ1.71 / 131.10 ms โ no change โ
โ QQuery 19 โ 42.28 / 42.99 ยฑ0.46 / 43.49 ms โ 42.55 / 44.22 ยฑ1.63 / 46.73 ms โ no change โ
โ QQuery 20 โ 36.30 / 37.10 ยฑ0.79 / 38.61 ms โ 36.74 / 38.13 ยฑ1.01 / 39.31 ms โ no change โ
โ QQuery 21 โ 18.00 / 18.30 ยฑ0.22 / 18.60 ms โ 18.16 / 18.37 ยฑ0.18 / 18.61 ms โ no change โ
โ QQuery 22 โ 67.16 / 69.22 ยฑ1.74 / 71.33 ms โ 65.12 / 66.87 ยฑ2.10 / 70.93 ms โ no change โ
โ QQuery 23 โ 375.83 / 383.63 ยฑ6.66 / 395.76 ms โ 360.62 / 376.45 ยฑ14.00 / 400.51 ms โ no change โ
โ QQuery 24 โ 226.97 / 230.70 ยฑ3.26 / 236.46 ms โ 241.25 / 247.70 ยฑ7.01 / 260.32 ms โ 1.07x slower โ
โ QQuery 25 โ 112.04 / 113.42 ยฑ1.09 / 115.02 ms โ 113.67 / 116.92 ยฑ3.67 / 123.09 ms โ no change โ
โ QQuery 26 โ 59.03 / 60.51 ยฑ1.67 / 63.63 ms โ 58.72 / 59.30 ยฑ0.40 / 59.92 ms โ no change โ
โ QQuery 27 โ 6.68 / 6.80 ยฑ0.08 / 6.91 ms โ 6.44 / 6.55 ยฑ0.13 / 6.80 ms โ no change โ
โ QQuery 28 โ 62.15 / 64.60 ยฑ2.85 / 70.13 ms โ 57.82 / 63.85 ยฑ5.61 / 74.49 ms โ no change โ
โ QQuery 29 โ 100.67 / 103.00 ยฑ1.86 / 105.29 ms โ 98.78 / 100.89 ยฑ2.64 / 105.99 ms โ no change โ
โ QQuery 30 โ 33.88 / 35.12 ยฑ1.06 / 37.04 ms โ 33.00 / 33.36 ยฑ0.31 / 33.89 ms โ +1.05x faster โ
โ QQuery 31 โ 115.23 / 116.70 ยฑ1.86 / 120.36 ms โ 113.27 / 114.44 ยฑ1.06 / 116.36 ms โ no change โ
โ QQuery 32 โ 21.21 / 21.63 ยฑ0.35 / 22.08 ms โ 21.43 / 21.77 ยฑ0.31 / 22.30 ms โ no change โ
โ QQuery 33 โ 38.11 / 39.22 ยฑ0.98 / 41.05 ms โ 38.50 / 39.05 ยฑ0.49 / 39.76 ms โ no change โ
โ QQuery 34 โ 10.09 / 10.39 ยฑ0.19 / 10.62 ms โ 10.20 / 10.28 ยฑ0.07 / 10.41 ms โ no change โ
โ QQuery 35 โ 74.58 / 75.04 ยฑ0.28 / 75.45 ms โ 75.31 / 77.71 ยฑ2.63 / 81.42 ms โ no change โ
โ QQuery 36 โ 6.00 / 6.11 ยฑ0.18 / 6.46 ms โ 5.98 / 6.14 ยฑ0.15 / 6.38 ms โ no change โ
โ QQuery 37 โ 7.09 / 7.26 ยฑ0.09 / 7.33 ms โ 7.01 / 7.07 ยฑ0.05 / 7.14 ms โ no change โ
โ QQuery 38 โ 64.21 / 65.26 ยฑ0.58 / 65.79 ms โ 65.19 / 66.28 ยฑ0.84 / 67.11 ms โ no change โ
โ QQuery 39 โ 93.70 / 97.18 ยฑ2.89 / 101.49 ms โ 91.58 / 93.07 ยฑ0.99 / 94.39 ms โ no change โ
โ QQuery 40 โ 25.10 / 25.33 ยฑ0.15 / 25.45 ms โ 23.93 / 24.90 ยฑ0.98 / 26.55 ms โ no change โ
โ QQuery 41 โ 12.18 / 12.45 ยฑ0.18 / 12.67 ms โ 12.11 / 12.28 ยฑ0.11 / 12.40 ms โ no change โ
โ QQuery 42 โ 24.98 / 25.47 ยฑ0.51 / 26.40 ms โ 24.84 / 25.47 ยฑ0.69 / 26.67 ms โ no change โ
โ QQuery 43 โ 5.46 / 5.52 ยฑ0.05 / 5.59 ms โ 5.11 / 5.22 ยฑ0.11 / 5.43 ms โ +1.06x faster โ
โ QQuery 44 โ 10.04 / 10.18 ยฑ0.12 / 10.40 ms โ 9.62 / 9.78 ยฑ0.12 / 9.94 ms โ no change โ
โ QQuery 45 โ 43.78 / 45.07 ยฑ0.96 / 45.96 ms โ 40.94 / 42.02 ยฑ0.97 / 43.47 ms โ +1.07x faster โ
โ QQuery 46 โ 12.42 / 12.84 ยฑ0.26 / 13.15 ms โ 12.22 / 12.47 ยฑ0.14 / 12.62 ms โ no change โ
โ QQuery 47 โ 253.54 / 267.82 ยฑ7.40 / 273.66 ms โ 241.33 / 244.39 ยฑ3.67 / 250.05 ms โ +1.10x faster โ
โ QQuery 48 โ 97.63 / 98.26 ยฑ0.46 / 98.91 ms โ 96.45 / 97.89 ยฑ1.09 / 99.41 ms โ no change โ
โ QQuery 49 โ 77.43 / 80.29 ยฑ2.96 / 84.19 ms โ 77.68 / 78.34 ยฑ0.73 / 79.65 ms โ no change โ
โ QQuery 50 โ 59.74 / 60.38 ยฑ0.43 / 61.08 ms โ 60.66 / 61.09 ยฑ0.28 / 61.41 ms โ no change โ
โ QQuery 51 โ 92.02 / 94.08 ยฑ1.99 / 97.33 ms โ 95.62 / 98.26 ยฑ3.21 / 104.41 ms โ no change โ
โ QQuery 52 โ 24.98 / 25.33 ยฑ0.37 / 26.03 ms โ 24.61 / 24.88 ยฑ0.16 / 25.09 ms โ no change โ
โ QQuery 53 โ 30.28 / 30.70 ยฑ0.24 / 30.99 ms โ 30.36 / 30.47 ยฑ0.09 / 30.63 ms โ no change โ
โ QQuery 54 โ 57.02 / 58.41 ยฑ0.76 / 59.29 ms โ 56.25 / 58.41 ยฑ3.03 / 64.41 ms โ no change โ
โ QQuery 55 โ 24.45 / 24.93 ยฑ0.66 / 26.23 ms โ 24.41 / 25.17 ยฑ0.49 / 25.86 ms โ no change โ
โ QQuery 56 โ 40.61 / 41.31 ยฑ0.62 / 42.11 ms โ 39.70 / 40.37 ยฑ0.46 / 41.03 ms โ no change โ
โ QQuery 57 โ 182.46 / 186.90 ยฑ2.32 / 189.12 ms โ 180.37 / 182.57 ยฑ2.61 / 186.63 ms โ no change โ
โ QQuery 58 โ 116.71 / 119.47 ยฑ2.92 / 124.73 ms โ 116.77 / 117.32 ยฑ0.36 / 117.89 ms โ no change โ
โ QQuery 59 โ 118.98 / 119.44 ยฑ0.29 / 119.76 ms โ 119.33 / 120.63 ยฑ1.69 / 123.89 ms โ no change โ
โ QQuery 60 โ 40.51 / 41.77 ยฑ1.13 / 43.72 ms โ 40.91 / 41.43 ยฑ0.45 / 42.05 ms โ no change โ
โ QQuery 61 โ 12.53 / 12.70 ยฑ0.16 / 12.94 ms โ 12.43 / 12.81 ยฑ0.26 / 13.25 ms โ no change โ
โ QQuery 62 โ 47.48 / 47.72 ยฑ0.15 / 47.93 ms โ 47.25 / 47.47 ยฑ0.14 / 47.65 ms โ no change โ
โ QQuery 63 โ 29.99 / 30.17 ยฑ0.20 / 30.55 ms โ 30.86 / 32.94 ยฑ3.15 / 39.16 ms โ 1.09x slower โ
โ QQuery 64 โ 414.85 / 422.61 ยฑ6.52 / 434.41 ms โ 417.23 / 426.30 ยฑ5.47 / 434.19 ms โ no change โ
โ QQuery 65 โ 152.89 / 155.26 ยฑ2.19 / 159.30 ms โ 146.82 / 149.19 ยฑ1.60 / 151.28 ms โ no change โ
โ QQuery 66 โ 82.85 / 83.31 ยฑ0.40 / 83.93 ms โ 83.39 / 84.66 ยฑ1.96 / 88.55 ms โ no change โ
โ QQuery 67 โ 259.76 / 266.18 ยฑ5.49 / 275.95 ms โ 254.56 / 260.49 ยฑ4.35 / 267.35 ms โ no change โ
โ QQuery 68 โ 12.41 / 12.57 ยฑ0.13 / 12.78 ms โ 12.10 / 13.27 ยฑ1.99 / 17.23 ms โ 1.06x slower โ
โ QQuery 69 โ 59.33 / 59.79 ยฑ0.52 / 60.67 ms โ 58.13 / 60.14 ยฑ3.03 / 66.17 ms โ no change โ
โ QQuery 70 โ 108.50 / 113.19 ยฑ5.64 / 123.60 ms โ 108.09 / 111.69 ยฑ3.61 / 118.54 ms โ no change โ
โ QQuery 71 โ 36.30 / 37.16 ยฑ0.77 / 38.44 ms โ 35.78 / 37.60 ยฑ2.79 / 43.15 ms โ no change โ
โ QQuery 72 โ 2225.88 / 2274.08 ยฑ45.60 / 2350.00 ms โ 2148.28 / 2257.24 ยฑ82.26 / 2360.63 ms โ no change โ
โ QQuery 73 โ 9.90 / 10.18 ยฑ0.21 / 10.53 ms โ 10.11 / 10.23 ยฑ0.09 / 10.36 ms โ no change โ
โ QQuery 74 โ 184.90 / 189.85 ยฑ4.31 / 197.85 ms โ 187.04 / 192.11 ยฑ6.28 / 203.82 ms โ no change โ
โ QQuery 75 โ 153.23 / 156.04 ยฑ4.60 / 165.22 ms โ 151.72 / 153.61 ยฑ1.73 / 155.63 ms โ no change โ
โ QQuery 76 โ 36.04 / 36.52 ยฑ0.35 / 37.09 ms โ 35.56 / 38.17 ยฑ3.76 / 45.65 ms โ no change โ
โ QQuery 77 โ 62.28 / 63.20 ยฑ0.71 / 64.46 ms โ 62.20 / 62.62 ยฑ0.36 / 63.04 ms โ no change โ
โ QQuery 78 โ 205.50 / 209.38 ยฑ3.91 / 216.54 ms โ 201.24 / 204.73 ยฑ3.09 / 208.98 ms โ no change โ
โ QQuery 79 โ 69.54 / 72.23 ยฑ3.95 / 80.07 ms โ 68.48 / 69.08 ยฑ0.46 / 69.84 ms โ no change โ
โ QQuery 80 โ 101.38 / 103.82 ยฑ1.81 / 106.04 ms โ 102.60 / 105.71 ยฑ3.77 / 112.44 ms โ no change โ
โ QQuery 81 โ 27.02 / 28.15 ยฑ1.67 / 31.45 ms โ 27.38 / 27.64 ยฑ0.27 / 28.14 ms โ no change โ
โ QQuery 82 โ 17.07 / 17.28 ยฑ0.17 / 17.46 ms โ 17.58 / 19.20 ยฑ3.06 / 25.32 ms โ 1.11x slower โ
โ QQuery 83 โ 41.35 / 43.84 ยฑ2.81 / 49.11 ms โ 41.26 / 42.14 ยฑ0.54 / 42.65 ms โ no change โ
โ QQuery 84 โ 31.13 / 31.85 ยฑ0.47 / 32.44 ms โ 30.90 / 31.47 ยฑ0.32 / 31.82 ms โ no change โ
โ QQuery 85 โ 107.34 / 108.73 ยฑ0.77 / 109.49 ms โ 108.33 / 111.18 ยฑ3.34 / 117.69 ms โ no change โ
โ QQuery 86 โ 25.69 / 27.81 ยฑ3.32 / 34.41 ms โ 25.59 / 26.32 ยฑ0.69 / 27.64 ms โ +1.06x faster โ
โ QQuery 87 โ 63.88 / 65.65 ยฑ1.35 / 67.64 ms โ 63.78 / 64.68 ยฑ0.87 / 66.18 ms โ no change โ
โ QQuery 88 โ 63.39 / 64.03 ยฑ0.40 / 64.63 ms โ 62.98 / 64.64 ยฑ2.46 / 69.51 ms โ no change โ
โ QQuery 89 โ 36.51 / 37.41 ยฑ1.10 / 39.51 ms โ 36.68 / 37.40 ยฑ0.47 / 38.10 ms โ no change โ
โ QQuery 90 โ 17.67 / 17.90 ยฑ0.16 / 18.14 ms โ 17.54 / 17.92 ยฑ0.55 / 19.01 ms โ no change โ
โ QQuery 91 โ 47.66 / 48.87 ยฑ1.32 / 51.30 ms โ 46.08 / 46.56 ยฑ0.43 / 47.34 ms โ no change โ
โ QQuery 92 โ 31.33 / 31.88 ยฑ0.33 / 32.23 ms โ 30.69 / 30.96 ยฑ0.20 / 31.27 ms โ no change โ
โ QQuery 93 โ 51.20 / 52.61 ยฑ1.32 / 54.75 ms โ 50.56 / 52.09 ยฑ2.04 / 55.94 ms โ no change โ
โ QQuery 94 โ 39.67 / 39.96 ยฑ0.22 / 40.31 ms โ 38.76 / 39.47 ยฑ0.48 / 40.14 ms โ no change โ
โ QQuery 95 โ 84.27 / 86.78 ยฑ2.66 / 90.60 ms โ 81.96 / 82.51 ยฑ0.35 / 83.04 ms โ no change โ
โ QQuery 96 โ 24.86 / 25.10 ยฑ0.25 / 25.51 ms โ 24.58 / 24.86 ยฑ0.26 / 25.32 ms โ no change โ
โ QQuery 97 โ 47.51 / 48.30 ยฑ0.49 / 48.87 ms โ 47.51 / 49.65 ยฑ1.85 / 52.71 ms โ no change โ
โ QQuery 98 โ 43.87 / 44.49 ยฑ0.69 / 45.44 ms โ 44.18 / 44.95 ยฑ0.76 / 46.40 ms โ no change โ
โ QQuery 99 โ 72.15 / 74.02 ยฑ3.09 / 80.18 ms โ 72.27 / 72.51 ยฑ0.25 / 72.96 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 10493.65ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 10356.46ms โ
โ Average Time (HEAD) โ 106.00ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 104.61ms โ
โ Queries Faster โ 9 โ
โ Queries Slower โ 4 โ
โ Queries with No Change โ 86 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.1 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 240.4s |
| CPU sys | 6.4s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.3 GiB |
| Avg memory | 1.6 GiB |
| CPU user | 235.9s |
| CPU sys | 6.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951360422
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951351156-1004-bc4hl 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5171425845
@alamb @andygrove @adriangbot
You might be interested in such a change :) Around 1.5-2x (or more) performance improvements on S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5184499823
I will review this PR later this week -- sorry for the dealy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951434436
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951427817-1007-2hn7p 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5297913079
Another one to consider: when the pushed down returns false for all values, we will skip the other requests (in which case it's more optimal than having wasted any IO).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496
run benchmarks
```
env:
SIMULATE_IO_LATENCY: 'true'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817
run benchmarks
```
env:
DATAFUSION_EXECUTION_PARQUET_PREFETCH_SIZE: "20971520"
SIMULATE_LATENCY: "true"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5197395369
FWIW the DuckDB folks posted a blog that describes a very similar implementation
https://duckdb.org/2026/07/31/asynchronous-io
The configuration is interesting -- you can bound by number of outstanding I/O or memory:
https://duckdb.org/2026/07/31/asynchronous-io#memory-management
I think we can do something similar, but the trick is how much memory to use for prefetching / buffering (compared to hash tables, etc)
(BTW I am still reading this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951771957
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951762927-1015-9dkvq 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951358348
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951351156-1006-kpn29 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951436467
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951427817-1008-n2rgl 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951769319
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951762927-1013-kpnxx 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5170383598
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23492?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `87.20379%` with `27 lines` in your changes missing coverage. Please review.
:warning: Please [upload](https://docs.codecov.com/docs/codecov-uploader) report for BASE (`main@2880e10`). [Learn more](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#section-missing-base-commit) about missing BASE report.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23492?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/proto-common/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&filepath=datafusion%2Fproto-common%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1jb21tb24vc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [15 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/logical\_plan/file\_formats.rs](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Flogical_plan%2Ffile_formats.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvbG9naWNhbF9wbGFuL2ZpbGVfZm9ybWF0cy5ycw==) | 0.00% | [5 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/datasource-parquet/src/push\_decoder.rs](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fpush_decoder.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3B1c2hfZGVjb2Rlci5ycw==) | 97.02% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23492?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23492 +/- ##
=======================================
Coverage ? 80.88%
=======================================
Files ? 1101
Lines ? 375974
Branches ? 375974
=======================================
Hits ? 304093
Misses ? 53773
Partials ? 18108
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23492?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-6374981193
FYI I tried to create a high level discussion of what we are trying to do here:
- https://github.com/apache/datafusion/issues/26682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156
run benchmarks
```
env:
SIMULATE_LATENCY: 'true'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951430639
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 106.75 / 173.34 ยฑ37.58 / 207.02 ms โ 106.76 / 172.96 ยฑ37.47 / 206.86 ms โ no change โ
โ QQuery 2 โ 523.34 / 653.53 ยฑ68.91 / 709.59 ms โ 521.03 / 651.03 ยฑ68.76 / 706.98 ms โ no change โ
โ QQuery 3 โ 382.82 / 429.78 ยฑ30.64 / 474.23 ms โ 383.78 / 429.70 ยฑ31.12 / 474.40 ms โ no change โ
โ QQuery 4 โ 1307.30 / 1348.36 ยฑ33.74 / 1405.97 ms โ 1128.54 / 1241.84 ยฑ89.85 / 1358.33 ms โ +1.09x faster โ
โ QQuery 5 โ 500.04 / 586.73 ยฑ61.37 / 682.91 ms โ 496.86 / 574.19 ยฑ64.98 / 680.18 ms โ no change โ
โ QQuery 6 โ 542.55 / 698.28 ยฑ103.70 / 820.55 ms โ 543.75 / 684.37 ยฑ118.33 / 819.79 ms โ no change โ
โ QQuery 7 โ 575.61 / 695.86 ยฑ73.30 / 781.00 ms โ 586.29 / 669.92 ยฑ61.26 / 774.39 ms โ no change โ
โ QQuery 8 โ 484.14 / 572.34 ยฑ57.85 / 641.07 ms โ 529.35 / 586.60 ยฑ53.95 / 652.53 ms โ no change โ
โ QQuery 9 โ 1381.94 / 1491.24 ยฑ89.37 / 1590.53 ms โ 1376.96 / 1488.69 ยฑ93.35 / 1592.71 ms โ no change โ
โ QQuery 10 โ 752.46 / 785.93 ยฑ34.36 / 849.70 ms โ 691.07 / 770.28 ยฑ49.15 / 842.33 ms โ no change โ
โ QQuery 11 โ 822.21 / 1040.47 ยฑ138.45 / 1180.01 ms โ 892.86 / 982.10 ยฑ83.48 / 1111.55 ms โ +1.06x faster โ
โ QQuery 12 โ 345.75 / 374.56 ยฑ27.20 / 419.96 ms โ 346.53 / 375.05 ยฑ26.62 / 418.58 ms โ no change โ
โ QQuery 13 โ 663.75 / 742.47 ยฑ45.89 / 800.75 ms โ 681.64 / 745.84 ยฑ40.68 / 793.69 ms โ no change โ
โ QQuery 14 โ 3998.60 / 4198.64 ยฑ108.96 / 4294.33 ms โ 3783.60 / 4157.23 ยฑ200.68 / 4367.37 ms โ no change โ
โ QQuery 15 โ 416.16 / 486.93 ยฑ43.41 / 537.63 ms โ 406.72 / 492.43 ยฑ52.15 / 548.72 ms โ no change โ
โ QQuery 16 โ 111.48 / 174.97 ยฑ36.51 / 208.42 ms โ 107.89 / 174.83 ยฑ38.43 / 211.94 ms โ no change โ
โ QQuery 17 โ 632.62 / 712.27 ยฑ43.95 / 760.51 ms โ 632.86 / 708.54 ยฑ42.10 / 756.67 ms โ no change โ
โ QQuery 18 โ 729.44 / 809.61 ยฑ61.87 / 917.23 ms โ 718.19 / 802.32 ยฑ61.16 / 906.29 ms โ no change โ
โ QQuery 19 โ 540.16 / 602.05 ยฑ53.45 / 677.89 ms โ 510.22 / 577.19 ยฑ63.09 / 685.19 ms โ no change โ
โ QQuery 20 โ 326.13 / 400.77 ยฑ41.74 / 451.89 ms โ 325.23 / 398.36 ยฑ40.71 / 447.01 ms โ no change โ
โ QQuery 21 โ 507.26 / 571.11 ยฑ48.31 / 628.59 ms โ 507.72 / 569.94 ยฑ49.75 / 630.60 ms โ no change โ
โ QQuery 22 โ 477.25 / 531.13 ยฑ32.47 / 577.32 ms โ 474.81 / 527.00 ยฑ31.30 / 570.34 ms โ no change โ
โ QQuery 23 โ 2573.77 / 2711.82 ยฑ112.94 / 2855.04 ms โ 2488.55 / 2786.91 ยฑ155.74 / 2908.15 ms โ no change โ
โ QQuery 24 โ 2191.39 / 2360.08 ยฑ108.85 / 2480.72 ms โ 2180.68 / 2352.33 ยฑ105.23 / 2470.35 ms โ no change โ
โ QQuery 25 โ 696.64 / 744.57 ยฑ40.70 / 806.29 ms โ 681.34 / 732.65 ยฑ33.75 / 768.74 ms โ no change โ
โ QQuery 26 โ 574.90 / 639.30 ยฑ47.64 / 707.14 ms โ 602.62 / 649.77 ยฑ33.25 / 700.96 ms โ no change โ
โ QQuery 27 โ 140.86 / 254.41 ยฑ84.35 / 359.02 ms โ 141.51 / 253.84 ยฑ83.72 / 357.77 ms โ no change โ
โ QQuery 28 โ 343.83 / 400.30 ยฑ32.99 / 434.79 ms โ 339.73 / 400.39 ยฑ33.75 / 433.23 ms โ no change โ
โ QQuery 29 โ 733.80 / 788.75 ยฑ42.69 / 837.74 ms โ 720.22 / 787.15 ยฑ43.86 / 823.65 ms โ no change โ
โ QQuery 30 โ 563.95 / 655.42 ยฑ68.93 / 750.50 ms โ 558.34 / 623.94 ยฑ67.67 / 744.57 ms โ no change โ
โ QQuery 31 โ 1135.41 / 1270.37 ยฑ80.66 / 1347.17 ms โ 1140.54 / 1223.70 ยฑ81.76 / 1322.81 ms โ no change โ
โ QQuery 32 โ 415.91 / 460.50 ยฑ34.09 / 490.68 ms โ 416.17 / 451.70 ยฑ31.20 / 489.95 ms โ no change โ
โ QQuery 33 โ 522.84 / 577.71 ยฑ38.72 / 643.22 ms โ 521.03 / 567.55 ยฑ33.59 / 611.24 ms โ no change โ
โ QQuery 34 โ 194.14 / 306.16 ยฑ59.81 / 359.42 ms โ 194.81 / 305.90 ยฑ59.30 / 357.93 ms โ no change โ
โ QQuery 35 โ 667.98 / 729.33 ยฑ43.90 / 804.30 ms โ 708.05 / 752.58 ยฑ32.11 / 800.31 ms โ no change โ
โ QQuery 36 โ 107.74 / 173.56 ยฑ37.51 / 207.44 ms โ 107.35 / 174.05 ยฑ37.94 / 207.47 ms โ no change โ
โ QQuery 37 โ 221.56 / 276.50 ยฑ30.25 / 310.63 ms โ 220.81 / 276.35 ยฑ30.41 / 309.79 ms โ no change โ
โ QQuery 38 โ 663.16 / 711.30 ยฑ58.49 / 819.44 ms โ 615.30 / 668.69 ยฑ42.65 / 738.32 ms โ +1.06x faster โ
โ QQuery 39 โ 1843.74 / 2010.46 ยฑ113.42 / 2150.12 ms โ 1838.56 / 2009.63 ยฑ119.14 / 2162.19 ms โ no change โ
โ QQuery 40 โ 468.55 / 494.68 ยฑ20.87 / 512.95 ms โ 467.59 / 493.87 ยฑ20.82 / 511.48 ms โ no change โ
โ QQuery 41 โ 178.33 / 279.93 ยฑ72.79 / 382.37 ms โ 176.36 / 279.08 ยฑ73.26 / 381.68 ms โ no change โ
โ QQuery 42 โ 308.57 / 404.74 ยฑ67.12 / 472.67 ms โ 309.42 / 404.43 ยฑ67.14 / 472.91 ms โ no change โ
โ QQuery 43 โ 106.39 / 153.38 ยฑ40.15 / 206.58 ms โ 106.61 / 153.69 ยฑ39.85 / 206.45 ms โ no change โ
โ QQuery 44 โ 196.92 / 262.49 ยฑ47.69 / 303.15 ms โ 196.50 / 262.27 ยฑ47.43 / 302.75 ms โ no change โ
โ QQuery 45 โ 459.55 / 547.71 ยฑ67.36 / 654.70 ms โ 459.15 / 529.23 ยฑ56.21 / 583.24 ms โ no change โ
โ QQuery 46 โ 290.03 / 346.04 ยฑ41.49 / 399.38 ms โ 291.04 / 345.56 ยฑ41.02 / 399.13 ms โ no change โ
โ QQuery 47 โ 1267.30 / 1308.68 ยฑ38.49 / 1360.00 ms โ 1237.86 / 1257.91 ยฑ30.96 / 1319.52 ms โ no change โ
โ QQuery 48 โ 600.38 / 706.21 ยฑ62.99 / 776.65 ms โ 598.78 / 704.04 ยฑ60.39 / 768.28 ms โ no change โ
โ QQuery 49 โ 583.80 / 633.48 ยฑ48.65 / 705.87 ms โ 580.02 / 631.01 ยฑ50.67 / 699.94 ms โ no change โ
โ QQuery 50 โ 540.78 / 634.74 ยฑ49.94 / 682.69 ms โ 541.20 / 636.56 ยฑ53.94 / 697.91 ms โ no change โ
โ QQuery 51 โ 524.47 / 562.00 ยฑ46.18 / 650.98 ms โ 523.45 / 542.31 ยฑ15.32 / 557.53 ms โ no change โ
โ QQuery 52 โ 314.15 / 406.04 ยฑ65.67 / 468.96 ms โ 307.83 / 405.83 ยฑ68.31 / 475.32 ms โ no change โ
โ QQuery 53 โ 413.59 / 453.64 ยฑ47.20 / 514.04 ms โ 412.24 / 454.18 ยฑ49.60 / 515.43 ms โ no change โ
โ QQuery 54 โ 905.83 / 961.61 ยฑ52.70 / 1041.02 ms โ 903.68 / 960.55 ยฑ49.01 / 1037.45 ms โ no change โ
โ QQuery 55 โ 378.44 / 423.36 ยฑ30.99 / 467.27 ms โ 379.82 / 423.24 ยฑ30.32 / 467.26 ms โ no change โ
โ QQuery 56 โ 588.75 / 615.81 ยฑ23.17 / 656.89 ms โ 553.01 / 612.21 ยฑ32.91 / 643.30 ms โ no change โ
โ QQuery 57 โ 1088.29 / 1189.51 ยฑ90.16 / 1308.40 ms โ 1060.72 / 1163.01 ยฑ98.46 / 1286.69 ms โ no change โ
โ QQuery 58 โ 1160.99 / 1328.67 ยฑ94.75 / 1422.42 ms โ 1160.02 / 1326.16 ยฑ94.39 / 1422.87 ms โ no change โ
โ QQuery 59 โ 895.59 / 930.38 ยฑ50.43 / 1030.73 ms โ 898.87 / 955.39 ยฑ61.45 / 1030.87 ms โ no change โ
โ QQuery 60 โ 566.68 / 653.88 ยฑ74.47 / 747.42 ms โ 568.71 / 636.42 ยฑ59.29 / 734.11 ms โ no change โ
โ QQuery 61 โ 393.51 / 395.41 ยฑ1.19 / 396.92 ms โ 394.55 / 394.89 ยฑ0.30 / 395.29 ms โ no change โ
โ QQuery 62 โ 580.57 / 601.16 ยฑ24.08 / 630.89 ms โ 580.02 / 600.49 ยฑ24.06 / 631.40 ms โ no change โ
โ QQuery 63 โ 416.01 / 440.68 ยฑ26.98 / 474.87 ms โ 417.89 / 440.94 ยฑ27.72 / 475.56 ms โ no change โ
โ QQuery 64 โ 3171.88 / 3335.48 ยฑ87.51 / 3413.91 ms โ 3220.62 / 3303.17 ยฑ67.19 / 3376.38 ms โ no change โ
โ QQuery 65 โ 635.04 / 741.32 ยฑ62.19 / 823.80 ms โ 634.46 / 740.26 ยฑ61.83 / 820.12 ms โ no change โ
โ QQuery 66 โ 571.00 / 598.88 ยฑ41.51 / 678.72 ms โ 558.71 / 597.90 ยฑ48.31 / 689.62 ms โ no change โ
โ QQuery 67 โ 637.05 / 704.61 ยฑ45.22 / 754.11 ms โ 595.47 / 689.45 ยฑ54.72 / 733.04 ms โ no change โ
โ QQuery 68 โ 309.89 / 323.44 ยฑ9.55 / 336.69 ms โ 310.62 / 323.87 ยฑ8.87 / 337.06 ms โ no change โ
โ QQuery 69 โ 749.31 / 801.00 ยฑ49.51 / 880.81 ms โ 704.18 / 820.98 ยฑ88.15 / 954.96 ms โ no change โ
โ QQuery 70 โ 715.28 / 768.91 ยฑ56.23 / 860.96 ms โ 690.92 / 763.67 ยฑ61.72 / 860.09 ms โ no change โ
โ QQuery 71 โ 522.68 / 573.91 ยฑ55.04 / 672.81 ms โ 523.51 / 595.40 ยฑ68.19 / 679.20 ms โ no change โ
โ QQuery 72 โ 3017.60 / 3168.44 ยฑ92.48 / 3255.38 ms โ 2945.80 / 3084.48 ยฑ80.87 / 3197.40 ms โ no change โ
โ QQuery 73 โ 194.46 / 309.15 ยฑ59.15 / 359.64 ms โ 192.15 / 278.35 ยฑ73.30 / 357.58 ms โ +1.11x faster โ
โ QQuery 74 โ 693.54 / 763.55 ยฑ56.51 / 828.92 ms โ 729.30 / 776.85 ยฑ33.79 / 827.24 ms โ no change โ
โ QQuery 75 โ 932.63 / 995.99 ยฑ44.20 / 1060.66 ms โ 937.74 / 988.08 ยฑ33.51 / 1040.74 ms โ no change โ
โ QQuery 76 โ 549.92 / 608.28 ยฑ47.56 / 660.29 ms โ 523.09 / 600.45 ยฑ55.82 / 656.35 ms โ no change โ
โ QQuery 77 โ 638.76 / 692.29 ยฑ58.84 / 804.53 ms โ 651.78 / 704.22 ยฑ42.22 / 758.08 ms โ no change โ
โ QQuery 78 โ 707.74 / 847.93 ยฑ76.19 / 938.32 ms โ 707.54 / 858.88 ยฑ80.31 / 944.82 ms โ no change โ
โ QQuery 79 โ 460.39 / 506.90 ยฑ42.58 / 583.86 ms โ 451.78 / 500.24 ยฑ45.23 / 583.28 ms โ no change โ
โ QQuery 80 โ 638.93 / 726.89 ยฑ68.10 / 790.74 ms โ 638.75 / 740.53 ยฑ58.92 / 802.35 ms โ no change โ
โ QQuery 81 โ 476.95 / 596.31 ยฑ67.09 / 680.93 ms โ 478.33 / 634.36 ยฑ83.95 / 722.79 ms โ 1.06x slower โ
โ QQuery 82 โ 555.60 / 644.23 ยฑ48.03 / 682.30 ms โ 558.70 / 644.30 ยฑ46.81 / 683.59 ms โ no change โ
โ QQuery 83 โ 941.23 / 997.82 ยฑ52.77 / 1098.23 ms โ 861.90 / 968.10 ยฑ61.73 / 1028.78 ms โ no change โ
โ QQuery 84 โ 438.29 / 501.11 ยฑ58.25 / 592.40 ms โ 438.16 / 503.37 ยฑ58.99 / 599.33 ms โ no change โ
โ QQuery 85 โ 817.07 / 889.11 ยฑ52.44 / 957.64 ms โ 822.30 / 889.58 ยฑ50.15 / 954.43 ms โ no change โ
โ QQuery 86 โ 247.76 / 314.80 ยฑ56.06 / 397.13 ms โ 251.27 / 315.04 ยฑ53.56 / 392.91 ms โ no change โ
โ QQuery 87 โ 534.33 / 661.04 ยฑ107.89 / 781.15 ms โ 534.76 / 652.94 ยฑ77.73 / 777.93 ms โ no change โ
โ QQuery 88 โ 737.17 / 738.57 ยฑ0.96 / 739.96 ms โ 737.51 / 739.59 ยฑ2.56 / 744.29 ms โ no change โ
โ QQuery 89 โ 426.32 / 459.26 ยฑ26.84 / 481.62 ms โ 425.59 / 459.47 ยฑ27.29 / 483.09 ms โ no change โ
โ QQuery 90 โ 426.22 / 465.76 ยฑ25.40 / 500.73 ms โ 427.04 / 467.07 ยฑ25.90 / 501.86 ms โ no change โ
โ QQuery 91 โ 523.90 / 663.48 ยฑ112.12 / 804.97 ms โ 524.25 / 664.97 ยฑ113.04 / 807.21 ms โ no change โ
โ QQuery 92 โ 370.32 / 439.80 ยฑ39.13 / 491.19 ms โ 372.43 / 418.17 ยฑ37.02 / 453.11 ms โ no change โ
โ QQuery 93 โ 464.34 / 480.01 ยฑ13.92 / 503.65 ms โ 468.25 / 481.80 ยฑ14.94 / 508.72 ms โ no change โ
โ QQuery 94 โ 480.16 / 531.16 ยฑ42.60 / 584.38 ms โ 478.75 / 528.74 ยฑ43.74 / 585.00 ms โ no change โ
โ QQuery 95 โ 693.34 / 778.02 ยฑ45.97 / 824.70 ms โ 689.94 / 777.65 ยฑ47.05 / 823.97 ms โ no change โ
โ QQuery 96 โ 407.36 / 448.70 ยฑ48.05 / 508.07 ms โ 406.81 / 447.36 ยฑ47.29 / 507.68 ms โ no change โ
โ QQuery 97 โ 502.55 / 558.67 ยฑ58.69 / 651.57 ms โ 440.81 / 512.15 ยฑ75.84 / 653.78 ms โ +1.09x faster โ
โ QQuery 98 โ 394.05 / 441.49 ยฑ31.31 / 483.61 ms โ 397.87 / 445.10 ยฑ30.52 / 485.60 ms โ no change โ
โ QQuery 99 โ 506.59 / 619.13 ยฑ64.68 / 693.66 ms โ 507.84 / 619.73 ยฑ64.60 / 696.32 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 75580.58ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 74947.09ms โ
โ Average Time (HEAD) โ 763.44ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 757.04ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 93 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 385.1s |
| Peak memory | 1.0 GiB |
| Avg memory | 534.2 MiB |
| CPU user | 243.6s |
| CPU sys | 12.0s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 1.0 GiB |
| Avg memory | 546.7 MiB |
| CPU user | 238.3s |
| CPU sys | 11.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951398154
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpch_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 479.77 / 578.09 ยฑ66.67 / 664.69 ms โ 480.98 / 563.31 ยฑ63.29 / 665.32 ms โ no change โ
โ QQuery 2 โ 869.75 / 900.47 ยฑ25.12 / 924.91 ms โ 868.59 / 917.48 ยฑ45.71 / 986.46 ms โ no change โ
โ QQuery 3 โ 997.01 / 1078.25 ยฑ44.22 / 1128.56 ms โ 922.05 / 1005.56 ยฑ70.50 / 1110.20 ms โ +1.07x faster โ
โ QQuery 4 โ 693.83 / 807.83 ยฑ70.93 / 904.82 ms โ 673.43 / 767.51 ยฑ79.95 / 882.60 ms โ no change โ
โ QQuery 5 โ 1064.07 / 1204.92 ยฑ110.42 / 1320.30 ms โ 1054.89 / 1260.78 ยฑ168.11 / 1495.01 ms โ no change โ
โ QQuery 6 โ 552.40 / 618.41 ยฑ40.98 / 681.45 ms โ 532.99 / 603.29 ยฑ65.42 / 726.94 ms โ no change โ
โ QQuery 7 โ 985.66 / 1124.43 ยฑ99.59 / 1256.48 ms โ 1003.43 / 1165.37 ยฑ127.60 / 1337.93 ms โ no change โ
โ QQuery 8 โ 1121.93 / 1278.46 ยฑ125.22 / 1504.58 ms โ 1138.03 / 1327.67 ยฑ153.70 / 1584.09 ms โ no change โ
โ QQuery 9 โ 1086.57 / 1229.57 ยฑ111.14 / 1396.04 ms โ 1129.25 / 1261.83 ยฑ140.25 / 1518.43 ms โ no change โ
โ QQuery 10 โ 886.67 / 972.76 ยฑ71.07 / 1086.68 ms โ 866.68 / 961.30 ยฑ64.00 / 1028.99 ms โ no change โ
โ QQuery 11 โ 758.53 / 832.44 ยฑ92.31 / 998.05 ms โ 717.49 / 811.78 ยฑ65.76 / 897.82 ms โ no change โ
โ QQuery 12 โ 742.56 / 800.17 ยฑ48.56 / 859.39 ms โ 746.31 / 823.43 ยฑ52.19 / 877.31 ms โ no change โ
โ QQuery 13 โ 404.31 / 443.49 ยฑ23.13 / 468.54 ms โ 399.76 / 440.77 ยฑ32.99 / 483.94 ms โ no change โ
โ QQuery 14 โ 564.86 / 673.59 ยฑ72.95 / 785.46 ms โ 597.19 / 698.79 ยฑ71.64 / 803.68 ms โ no change โ
โ QQuery 15 โ 1152.59 / 1249.81 ยฑ68.72 / 1357.13 ms โ 1147.63 / 1241.72 ยฑ66.30 / 1332.32 ms โ no change โ
โ QQuery 16 โ 325.71 / 423.52 ยฑ80.66 / 549.14 ms โ 325.89 / 390.94 ยฑ33.10 / 416.04 ms โ +1.08x faster โ
โ QQuery 17 โ 1086.15 / 1285.10 ยฑ126.96 / 1469.75 ms โ 1084.88 / 1230.63 ยฑ96.64 / 1382.79 ms โ no change โ
โ QQuery 18 โ 1320.64 / 1501.35 ยฑ93.37 / 1590.61 ms โ 1421.99 / 1535.79 ยฑ101.52 / 1659.19 ms โ no change โ
โ QQuery 19 โ 584.26 / 668.49 ยฑ55.29 / 740.05 ms โ 568.22 / 628.94 ยฑ34.71 / 673.08 ms โ +1.06x faster โ
โ QQuery 20 โ 1001.82 / 1062.33 ยฑ53.74 / 1145.33 ms โ 916.26 / 1026.74 ยฑ76.37 / 1140.60 ms โ no change โ
โ QQuery 21 โ 2002.19 / 2076.32 ยฑ67.95 / 2202.92 ms โ 2024.62 / 2234.73 ยฑ158.29 / 2460.76 ms โ 1.08x slower โ
โ QQuery 22 โ 513.82 / 560.19 ยฑ33.21 / 601.83 ms โ 524.30 / 553.38 ยฑ19.45 / 584.46 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 21370.03ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 21451.73ms โ
โ Average Time (HEAD) โ 971.36ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 975.08ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 18 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 837.0 MiB |
| Avg memory | 411.3 MiB |
| CPU user | 30.4s |
| CPU sys | 3.6s |
| Peak spill | 0 B |
**tpch โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 814.3 MiB |
| Avg memory | 420.6 MiB |
| CPU user | 30.3s |
| CPU sys | 3.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951475065
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpch_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 517.84 / 577.10 ยฑ50.58 / 643.57 ms โ 211.96 / 226.97 ยฑ9.68 / 238.87 ms โ +2.54x faster โ
โ QQuery 2 โ 870.15 / 905.54 ยฑ19.99 / 925.84 ms โ 651.23 / 734.80 ยฑ57.90 / 798.91 ms โ +1.23x faster โ
โ QQuery 3 โ 902.74 / 1024.11 ยฑ85.66 / 1112.70 ms โ 509.18 / 559.06 ยฑ36.85 / 606.43 ms โ +1.83x faster โ
โ QQuery 4 โ 784.11 / 826.71 ยฑ45.88 / 915.11 ms โ 397.51 / 402.62 ยฑ8.13 / 418.79 ms โ +2.05x faster โ
โ QQuery 5 โ 1081.22 / 1211.12 ยฑ87.80 / 1344.85 ms โ 707.30 / 777.56 ยฑ42.82 / 832.47 ms โ +1.56x faster โ
โ QQuery 6 โ 528.82 / 572.60 ยฑ52.23 / 668.77 ms โ 196.44 / 207.11 ยฑ8.48 / 214.11 ms โ +2.76x faster โ
โ QQuery 7 โ 1022.18 / 1177.67 ยฑ123.30 / 1339.96 ms โ 603.47 / 771.48 ยฑ98.70 / 886.04 ms โ +1.53x faster โ
โ QQuery 8 โ 1143.26 / 1284.65 ยฑ125.69 / 1501.18 ms โ 838.64 / 877.50 ยฑ20.78 / 899.98 ms โ +1.46x faster โ
โ QQuery 9 โ 1207.19 / 1267.35 ยฑ38.18 / 1320.28 ms โ 836.22 / 857.64 ยฑ19.65 / 886.92 ms โ +1.48x faster โ
โ QQuery 10 โ 846.80 / 940.00 ยฑ78.14 / 1067.06 ms โ 519.85 / 550.40 ยฑ26.86 / 590.87 ms โ +1.71x faster โ
โ QQuery 11 โ 702.82 / 834.58 ยฑ83.73 / 902.45 ms โ 616.14 / 692.04 ยฑ44.30 / 753.16 ms โ +1.21x faster โ
โ QQuery 12 โ 720.05 / 839.08 ยฑ69.01 / 930.83 ms โ 400.48 / 405.38 ยฑ7.68 / 420.65 ms โ +2.07x faster โ
โ QQuery 13 โ 401.34 / 410.46 ยฑ12.36 / 434.45 ms โ 319.66 / 364.57 ยฑ37.40 / 414.03 ms โ +1.13x faster โ
โ QQuery 14 โ 640.00 / 705.61 ยฑ65.75 / 821.83 ms โ 232.00 / 326.13 ยฑ57.74 / 405.80 ms โ +2.16x faster โ
โ QQuery 15 โ 1095.10 / 1240.39 ยฑ105.32 / 1369.28 ms โ 440.01 / 541.29 ยฑ82.00 / 616.86 ms โ +2.29x faster โ
โ QQuery 16 โ 358.91 / 410.30 ยฑ47.75 / 479.39 ms โ 264.25 / 327.11 ยฑ58.98 / 396.35 ms โ +1.25x faster โ
โ QQuery 17 โ 1083.62 / 1230.32 ยฑ78.95 / 1319.39 ms โ 510.97 / 568.71 ยฑ40.03 / 633.36 ms โ +2.16x faster โ
โ QQuery 18 โ 1431.65 / 1552.91 ยฑ68.27 / 1626.25 ms โ 799.67 / 803.02 ยฑ2.82 / 808.09 ms โ +1.93x faster โ
โ QQuery 19 โ 618.32 / 670.67 ยฑ40.47 / 717.47 ms โ 293.37 / 347.77 ยฑ39.60 / 415.20 ms โ +1.93x faster โ
โ QQuery 20 โ 985.73 / 1076.16 ยฑ64.94 / 1176.67 ms โ 529.80 / 568.83 ยฑ30.99 / 612.27 ms โ +1.89x faster โ
โ QQuery 21 โ 2043.66 / 2163.86 ยฑ141.43 / 2391.79 ms โ 948.09 / 953.00 ยฑ3.99 / 958.58 ms โ +2.27x faster โ
โ QQuery 22 โ 497.18 / 556.86 ยฑ56.29 / 652.88 ms โ 496.39 / 498.38 ยฑ2.02 / 502.18 ms โ +1.12x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 21478.03ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 12361.38ms โ
โ Average Time (HEAD) โ 976.27ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 561.88ms โ
โ Queries Faster โ 22 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 0 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 857.4 MiB |
| Avg memory | 418.9 MiB |
| CPU user | 31.2s |
| CPU sys | 3.7s |
| Peak spill | 0 B |
**tpch โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 65.0s |
| Peak memory | 925.4 MiB |
| Avg memory | 452.7 MiB |
| CPU user | 30.2s |
| CPU sys | 3.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5273121874
I have been thinking about this PR and how it could play into our "turn pushdown on by default" story
We have put a lot of work in upstream to make applying filters during parquet decoding faster (thanks to @hhhizzz @haohuaijin and many others). However, enabling filter pushdown today **also** changes the I/O patterns (so that the all the needed columns are not fetched up front, but instead are fetched sequentially)
I think one thing we could do with a nice I/O pre-fetching abstraction would be to start a request to fetch bytes for all columns required in a single request, so the I/O pattern is the same -- and then just enable the pushdown filter application once we already had the data ๐ค
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951185086
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-common v54.0.0 (current)
Built [ 35.865s] (current)
Parsing datafusion-common v54.0.0 (current)
Parsed [ 0.059s] (current)
Building datafusion-common v54.0.0 (baseline)
Built [ 34.527s] (baseline)
Parsing datafusion-common v54.0.0 (baseline)
Parsed [ 0.061s] (baseline)
Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.713s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ParquetOptions.prefetch_size in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1103
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 73.784s] datafusion-common
Building datafusion-datasource-parquet v54.0.0 (current)
Built [ 44.590s] (current)
Parsing datafusion-datasource-parquet v54.0.0 (current)
Parsed [ 0.031s] (current)
Building datafusion-datasource-parquet v54.0.0 (baseline)
Built [ 44.336s] (baseline)
Parsing datafusion-datasource-parquet v54.0.0 (baseline)
Parsed [ 0.031s] (baseline)
Checking datafusion-datasource-parquet v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.148s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 90.281s] datafusion-datasource-parquet
Building datafusion-proto v54.0.0 (current)
Built [ 60.972s] (current)
Parsing datafusion-proto v54.0.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-proto v54.0.0 (baseline)
Built [ 61.071s] (baseline)
Parsing datafusion-proto v54.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.258s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 124.169s] datafusion-proto
Building datafusion-proto-common v54.0.0 (current)
Built [ 21.787s] (current)
Parsing datafusion-proto-common v54.0.0 (current)
Parsed [ 0.048s] (current)
Building datafusion-proto-common v54.0.0 (baseline)
Built [ 21.809s] (baseline)
Parsing datafusion-proto-common v54.0.0 (baseline)
Parsed [ 0.049s] (baseline)
Checking datafusion-proto-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 1.140s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ParquetOptions.prefetch_size_opt in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:908
field ParquetOptions.prefetch_size_opt in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:908
field ParquetOptions.prefetch_size_opt in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:908
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 46.342s] datafusion-proto-common
Building datafusion-proto-models v54.0.0 (current)
Built [ 25.179s] (current)
Parsing datafusion-proto-models v54.0.0 (current)
Parsed [ 0.130s] (current)
Building datafusion-proto-models v54.0.0 (baseline)
Built [ 25.033s] (baseline)
Parsing datafusion-proto-models v54.0.0 (baseline)
Parsed [ 0.139s] (baseline)
Checking datafusion-proto-models v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 1.774s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ParquetOptions.prefetch_size_opt in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:908
field ParquetOptions.prefetch_size_opt in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/datafusion_proto_common.rs:908
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 53.270s] datafusion-proto-models
Building datafusion-sqllogictest v54.0.0 (current)
Built [ 185.421s] (current)
Parsing datafusion-sqllogictest v54.0.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-sqllogictest v54.0.0 (baseline)
Built [ 185.314s] (baseline)
Parsing datafusion-sqllogictest v54.0.0 (baseline)
Parsed [ 0.022s] (baseline)
Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.089s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 375.084s] datafusion-sqllogictest
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5191156983
Also https://github.com/apache/datafusion/pull/24086 from @adriangb has some interesting experiments.
Some other thoughts:
* The interaction with filter pushdown is interesting - what is the best way to avoid serialization behind a large number of row filters while also minimizing bandwidth / decoding usage? The current implementation just plans the filters serially (`f1 -> f2 -> f3 -> f4 -> ... -> full projection`) without considering IO latency.
* Should we just configure a fixed budget or should we do something with bandwidth?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5174919973
What can we expect for memory consumption? Will we use more memory because we prefetch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5203406850
I agree, it would be good to spend some cycles iterating on the best design forward that fits DataFusion well and we can keep for longer (and enable by default).
I think it would be interesting as well to test on block storage (EBS) as well if we see any (smaller) benefit there (this would benefit official ClickBench benchmark AFAIK) (having latency profile of around ~1-10ms).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5298317419
> I think for ClickBench it might not matter (no offset index?) - so perhaps we should special case this (just request the columns).
Should we make a ClickBench dataset with a page index? Or an option on the CB benchmark if possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951267861
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.21 / 3.89 ยฑ5.28 / 14.45 ms โ 1.22 / 3.92 ยฑ5.36 / 14.64 ms โ no change โ
โ QQuery 1 โ 12.67 / 12.82 ยฑ0.12 / 13.00 ms โ 12.80 / 13.29 ยฑ0.75 / 14.77 ms โ no change โ
โ QQuery 2 โ 36.35 / 36.67 ยฑ0.28 / 37.17 ms โ 36.17 / 36.45 ยฑ0.21 / 36.81 ms โ no change โ
โ QQuery 3 โ 30.89 / 32.87 ยฑ1.78 / 35.14 ms โ 30.91 / 31.44 ยฑ0.54 / 32.37 ms โ no change โ
โ QQuery 4 โ 220.43 / 223.77 ยฑ2.29 / 226.98 ms โ 223.27 / 224.33 ยฑ0.77 / 225.32 ms โ no change โ
โ QQuery 5 โ 269.28 / 271.62 ยฑ1.31 / 272.85 ms โ 267.46 / 269.61 ยฑ1.82 / 272.88 ms โ no change โ
โ QQuery 6 โ 1.28 / 1.43 ยฑ0.21 / 1.85 ms โ 1.24 / 1.39 ยฑ0.22 / 1.82 ms โ no change โ
โ QQuery 7 โ 14.01 / 14.09 ยฑ0.07 / 14.20 ms โ 14.00 / 14.15 ยฑ0.11 / 14.31 ms โ no change โ
โ QQuery 8 โ 320.68 / 365.81 ยฑ25.33 / 390.02 ms โ 319.66 / 323.54 ยฑ2.97 / 328.67 ms โ +1.13x faster โ
โ QQuery 9 โ 431.87 / 457.11 ยฑ16.12 / 482.57 ms โ 452.25 / 456.93 ยฑ5.72 / 467.54 ms โ no change โ
โ QQuery 10 โ 68.84 / 69.51 ยฑ0.35 / 69.82 ms โ 68.89 / 73.03 ยฑ6.62 / 86.18 ms โ 1.05x slower โ
โ QQuery 11 โ 80.37 / 81.61 ยฑ1.01 / 83.12 ms โ 80.38 / 81.90 ยฑ1.05 / 82.91 ms โ no change โ
โ QQuery 12 โ 263.40 / 266.92 ยฑ3.40 / 273.29 ms โ 266.44 / 273.63 ยฑ8.26 / 287.55 ms โ no change โ
โ QQuery 13 โ 364.96 / 379.14 ยฑ9.92 / 393.38 ms โ 369.41 / 387.44 ยฑ13.93 / 403.78 ms โ no change โ
โ QQuery 14 โ 278.05 / 287.49 ยฑ12.24 / 311.71 ms โ 279.72 / 291.50 ยฑ15.77 / 322.44 ms โ no change โ
โ QQuery 15 โ 263.70 / 271.44 ยฑ4.77 / 275.85 ms โ 268.89 / 275.73 ยฑ6.71 / 287.98 ms โ no change โ
โ QQuery 16 โ 598.22 / 608.12 ยฑ7.46 / 621.32 ms โ 608.54 / 615.16 ยฑ5.83 / 625.66 ms โ no change โ
โ QQuery 17 โ 603.73 / 617.45 ยฑ9.54 / 630.51 ms โ 613.25 / 624.74 ยฑ11.07 / 641.08 ms โ no change โ
โ QQuery 18 โ 1221.59 / 1261.77 ยฑ21.63 / 1286.49 ms โ 1219.92 / 1255.19 ยฑ24.97 / 1293.43 ms โ no change โ
โ QQuery 19 โ 27.56 / 28.42 ยฑ0.63 / 29.19 ms โ 28.00 / 28.21 ยฑ0.22 / 28.61 ms โ no change โ
โ QQuery 20 โ 515.01 / 523.51 ยฑ9.34 / 540.64 ms โ 513.61 / 516.83 ยฑ2.32 / 520.53 ms โ no change โ
โ QQuery 21 โ 511.12 / 517.58 ยฑ4.22 / 524.06 ms โ 513.50 / 517.54 ยฑ3.03 / 522.61 ms โ no change โ
โ QQuery 22 โ 979.06 / 993.99 ยฑ12.13 / 1015.83 ms โ 980.40 / 988.84 ยฑ7.59 / 1002.71 ms โ no change โ
โ QQuery 23 โ 3011.81 / 3051.66 ยฑ28.86 / 3099.32 ms โ 3011.24 / 3051.70 ยฑ35.25 / 3106.51 ms โ no change โ
โ QQuery 24 โ 40.67 / 40.94 ยฑ0.19 / 41.20 ms โ 40.94 / 41.56 ยฑ1.00 / 43.56 ms โ no change โ
โ QQuery 25 โ 110.58 / 114.32 ยฑ5.97 / 126.22 ms โ 110.25 / 119.23 ยฑ16.00 / 151.17 ms โ no change โ
โ QQuery 26 โ 41.33 / 44.02 ยฑ3.94 / 51.76 ms โ 40.96 / 42.87 ยฑ2.51 / 47.81 ms โ no change โ
โ QQuery 27 โ 662.85 / 668.40 ยฑ3.34 / 671.93 ms โ 660.34 / 670.54 ยฑ6.06 / 678.49 ms โ no change โ
โ QQuery 28 โ 3007.34 / 3050.63 ยฑ35.45 / 3100.53 ms โ 2991.50 / 3014.73 ยฑ16.29 / 3037.21 ms โ no change โ
โ QQuery 29 โ 40.66 / 44.57 ยฑ6.82 / 58.15 ms โ 40.95 / 52.06 ยฑ21.74 / 95.54 ms โ 1.17x slower โ
โ QQuery 30 โ 304.19 / 309.92 ยฑ4.59 / 318.12 ms โ 302.92 / 309.20 ยฑ8.72 / 326.30 ms โ no change โ
โ QQuery 31 โ 288.47 / 295.12 ยฑ7.33 / 309.07 ms โ 283.28 / 297.17 ยฑ14.18 / 323.74 ms โ no change โ
โ QQuery 32 โ 917.30 / 939.88 ยฑ12.68 / 955.45 ms โ 949.48 / 971.35 ยฑ24.77 / 1018.78 ms โ no change โ
โ QQuery 33 โ 1448.17 / 1474.86 ยฑ24.71 / 1517.37 ms โ 1443.13 / 1463.62 ยฑ12.49 / 1477.68 ms โ no change โ
โ QQuery 34 โ 1442.70 / 1461.85 ยฑ14.63 / 1477.20 ms โ 1450.73 / 1467.31 ยฑ12.93 / 1488.01 ms โ no change โ
โ QQuery 35 โ 274.23 / 283.47 ยฑ8.09 / 296.75 ms โ 271.38 / 292.73 ยฑ26.24 / 339.78 ms โ no change โ
โ QQuery 36 โ 64.61 / 78.06 ยฑ11.69 / 93.97 ms โ 66.90 / 70.75 ยฑ2.48 / 74.48 ms โ +1.10x faster โ
โ QQuery 37 โ 35.06 / 37.98 ยฑ3.43 / 43.91 ms โ 35.02 / 39.13 ยฑ2.61 / 41.59 ms โ no change โ
โ QQuery 38 โ 40.33 / 41.99 ยฑ1.77 / 44.44 ms โ 40.30 / 44.13 ยฑ2.80 / 48.93 ms โ 1.05x slower โ
โ QQuery 39 โ 139.84 / 150.55 ยฑ9.52 / 164.38 ms โ 139.70 / 145.87 ยฑ6.79 / 157.82 ms โ no change โ
โ QQuery 40 โ 13.63 / 16.08 ยฑ4.19 / 24.44 ms โ 13.71 / 15.64 ยฑ3.16 / 21.95 ms โ no change โ
โ QQuery 41 โ 13.57 / 13.64 ยฑ0.07 / 13.74 ms โ 13.60 / 14.42 ยฑ1.30 / 16.98 ms โ 1.06x slower โ
โ QQuery 42 โ 12.92 / 14.15 ยฑ2.25 / 18.64 ms โ 12.87 / 13.23 ยฑ0.24 / 13.53 ms โ +1.07x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 19459.12ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 19442.02ms โ
โ Average Time (HEAD) โ 452.54ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 452.14ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 4 โ
โ Queries with No Change โ 36 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 11.0 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 1003.2s |
| CPU sys | 67.5s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 11.4 GiB |
| Avg memory | 4.2 GiB |
| CPU user | 999.5s |
| CPU sys | 68.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951846782
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 106.46 / 173.02 ยฑ37.70 / 207.17 ms โ 106.62 / 172.84 ยฑ37.73 / 207.03 ms โ no change โ
โ QQuery 2 โ 512.17 / 649.37 ยฑ72.17 / 706.23 ms โ 520.41 / 650.36 ยฑ70.15 / 706.73 ms โ no change โ
โ QQuery 3 โ 382.08 / 427.22 ยฑ31.86 / 474.42 ms โ 382.53 / 429.64 ยฑ32.22 / 475.25 ms โ no change โ
โ QQuery 4 โ 1127.00 / 1271.27 ยฑ73.97 / 1324.92 ms โ 1199.36 / 1257.81 ยฑ42.96 / 1308.72 ms โ no change โ
โ QQuery 5 โ 499.24 / 572.13 ยฑ64.22 / 678.20 ms โ 501.02 / 582.53 ยฑ61.11 / 687.24 ms โ no change โ
โ QQuery 6 โ 610.02 / 715.94 ยฑ86.66 / 820.80 ms โ 535.95 / 701.22 ยฑ108.14 / 820.67 ms โ no change โ
โ QQuery 7 โ 595.60 / 692.52 ยฑ57.35 / 754.06 ms โ 595.18 / 696.20 ยฑ57.75 / 759.28 ms โ no change โ
โ QQuery 8 โ 571.17 / 620.00 ยฑ28.46 / 651.69 ms โ 536.56 / 605.87 ยฑ44.45 / 655.30 ms โ no change โ
โ QQuery 9 โ 1378.71 / 1488.86 ยฑ91.19 / 1590.00 ms โ 1377.87 / 1492.37 ยฑ94.70 / 1592.12 ms โ no change โ
โ QQuery 10 โ 696.01 / 792.23 ยฑ64.55 / 879.34 ms โ 687.82 / 764.24 ยฑ53.86 / 843.21 ms โ no change โ
โ QQuery 11 โ 822.97 / 996.86 ยฑ113.92 / 1138.17 ms โ 919.53 / 1015.79 ยฑ67.15 / 1092.81 ms โ no change โ
โ QQuery 12 โ 348.35 / 373.29 ยฑ25.24 / 417.51 ms โ 350.50 / 373.86 ยฑ24.76 / 418.28 ms โ no change โ
โ QQuery 13 โ 661.66 / 740.18 ยฑ45.96 / 790.37 ms โ 687.86 / 746.73 ยฑ39.16 / 797.30 ms โ no change โ
โ QQuery 14 โ 3685.00 / 4002.01 ยฑ174.29 / 4209.75 ms โ 3821.54 / 4135.10 ยฑ219.09 / 4419.67 ms โ no change โ
โ QQuery 15 โ 408.92 / 499.40 ยฑ48.91 / 549.28 ms โ 411.73 / 507.26 ยฑ62.08 / 582.23 ms โ no change โ
โ QQuery 16 โ 107.40 / 174.72 ยฑ38.51 / 210.86 ms โ 108.12 / 174.73 ยฑ38.35 / 210.63 ms โ no change โ
โ QQuery 17 โ 626.97 / 704.89 ยฑ44.77 / 756.82 ms โ 627.71 / 709.19 ยฑ45.39 / 758.83 ms โ no change โ
โ QQuery 18 โ 718.02 / 803.99 ยฑ61.11 / 907.82 ms โ 715.09 / 805.10 ยฑ61.43 / 905.24 ms โ no change โ
โ QQuery 19 โ 466.23 / 577.04 ยฑ66.27 / 643.79 ms โ 531.85 / 596.96 ยฑ56.47 / 683.26 ms โ no change โ
โ QQuery 20 โ 326.71 / 397.93 ยฑ39.82 / 446.05 ms โ 326.76 / 399.39 ยฑ39.99 / 445.76 ms โ no change โ
โ QQuery 21 โ 508.24 / 570.39 ยฑ48.32 / 627.75 ms โ 505.64 / 570.52 ยฑ49.82 / 633.34 ms โ no change โ
โ QQuery 22 โ 477.60 / 516.17 ยฑ21.02 / 541.48 ms โ 474.87 / 528.96 ยฑ32.65 / 570.98 ms โ no change โ
โ QQuery 23 โ 2495.22 / 2723.94 ยฑ147.71 / 2891.86 ms โ 2432.43 / 2711.02 ยฑ152.23 / 2860.42 ms โ no change โ
โ QQuery 24 โ 2192.55 / 2352.81 ยฑ102.38 / 2468.57 ms โ 2185.65 / 2357.55 ยฑ110.15 / 2483.78 ms โ no change โ
โ QQuery 25 โ 686.18 / 728.57 ยฑ35.45 / 777.66 ms โ 684.78 / 732.06 ยฑ33.89 / 777.28 ms โ no change โ
โ QQuery 26 โ 610.10 / 649.61 ยฑ33.45 / 705.15 ms โ 603.33 / 633.33 ยฑ21.57 / 669.76 ms โ no change โ
โ QQuery 27 โ 140.19 / 253.79 ยฑ84.30 / 358.70 ms โ 140.30 / 253.30 ยฑ84.35 / 358.82 ms โ no change โ
โ QQuery 28 โ 339.09 / 399.49 ยฑ33.51 / 432.17 ms โ 342.58 / 398.10 ยฑ32.36 / 433.31 ms โ no change โ
โ QQuery 29 โ 724.44 / 780.89 ยฑ39.29 / 823.42 ms โ 729.67 / 785.85 ยฑ42.76 / 834.89 ms โ no change โ
โ QQuery 30 โ 487.18 / 617.17 ยฑ91.76 / 740.43 ms โ 514.54 / 625.58 ยฑ83.70 / 743.63 ms โ no change โ
โ QQuery 31 โ 1192.95 / 1266.50 ยฑ58.80 / 1342.18 ms โ 1134.15 / 1262.18 ยฑ107.89 / 1405.83 ms โ no change โ
โ QQuery 32 โ 413.79 / 443.57 ยฑ26.85 / 487.62 ms โ 372.95 / 433.79 ยฑ50.77 / 487.67 ms โ no change โ
โ QQuery 33 โ 540.55 / 617.52 ยฑ55.74 / 703.32 ms โ 522.54 / 585.54 ยฑ39.85 / 643.33 ms โ +1.05x faster โ
โ QQuery 34 โ 193.47 / 304.98 ยฑ59.67 / 359.86 ms โ 194.23 / 306.29 ยฑ59.81 / 358.35 ms โ no change โ
โ QQuery 35 โ 684.68 / 739.15 ยฑ42.04 / 803.13 ms โ 723.34 / 751.07 ยฑ29.29 / 801.66 ms โ no change โ
โ QQuery 36 โ 108.08 / 173.33 ยฑ37.50 / 207.36 ms โ 107.56 / 173.41 ยฑ37.61 / 207.83 ms โ no change โ
โ QQuery 37 โ 221.67 / 276.34 ยฑ30.19 / 310.39 ms โ 222.05 / 276.33 ยฑ29.98 / 310.24 ms โ no change โ
โ QQuery 38 โ 602.18 / 665.90 ยฑ33.89 / 700.49 ms โ 598.40 / 667.90 ยฑ77.51 / 816.02 ms โ no change โ
โ QQuery 39 โ 1849.20 / 2008.00 ยฑ112.90 / 2142.61 ms โ 1841.51 / 2006.02 ยฑ111.28 / 2141.24 ms โ no change โ
โ QQuery 40 โ 464.51 / 492.49 ยฑ22.73 / 512.50 ms โ 467.79 / 493.57 ยฑ20.86 / 511.99 ms โ no change โ
โ QQuery 41 โ 176.30 / 279.08 ยฑ73.21 / 381.09 ms โ 175.16 / 278.59 ยฑ73.59 / 381.23 ms โ no change โ
โ QQuery 42 โ 313.82 / 404.78 ยฑ64.46 / 464.30 ms โ 313.95 / 404.34 ยฑ65.01 / 472.85 ms โ no change โ
โ QQuery 43 โ 106.71 / 153.28 ยฑ40.84 / 206.41 ms โ 106.75 / 152.93 ยฑ40.18 / 206.42 ms โ no change โ
โ QQuery 44 โ 198.42 / 262.14 ยฑ47.05 / 302.33 ms โ 197.42 / 261.93 ยฑ47.23 / 302.38 ms โ no change โ
โ QQuery 45 โ 416.40 / 541.07 ยฑ82.20 / 646.27 ms โ 451.73 / 526.45 ยฑ48.13 / 578.98 ms โ no change โ
โ QQuery 46 โ 289.65 / 345.37 ยฑ41.26 / 398.87 ms โ 309.42 / 351.92 ยฑ34.87 / 399.62 ms โ no change โ
โ QQuery 47 โ 1227.52 / 1257.84 ยฑ34.23 / 1323.00 ms โ 1230.91 / 1274.29 ยฑ35.29 / 1321.20 ms โ no change โ
โ QQuery 48 โ 585.05 / 704.80 ยฑ65.83 / 771.69 ms โ 606.65 / 707.91 ยฑ62.24 / 776.07 ms โ no change โ
โ QQuery 49 โ 579.54 / 629.86 ยฑ50.21 / 706.35 ms โ 522.95 / 603.49 ยฑ61.66 / 712.86 ms โ no change โ
โ QQuery 50 โ 526.90 / 635.40 ยฑ57.04 / 689.66 ms โ 541.26 / 635.29 ยฑ50.21 / 683.07 ms โ no change โ
โ QQuery 51 โ 504.35 / 540.06 ยฑ27.21 / 563.52 ms โ 522.92 / 542.69 ยฑ15.34 / 559.04 ms โ no change โ
โ QQuery 52 โ 313.96 / 403.67 ยฑ63.80 / 463.39 ms โ 314.78 / 405.64 ยฑ64.87 / 471.49 ms โ no change โ
โ QQuery 53 โ 407.93 / 453.22 ยฑ50.83 / 515.99 ms โ 409.80 / 453.29 ยฑ47.74 / 514.03 ms โ no change โ
โ QQuery 54 โ 905.92 / 946.53 ยฑ46.20 / 1033.73 ms โ 902.46 / 937.26 ยฑ34.75 / 997.29 ms โ no change โ
โ QQuery 55 โ 377.06 / 423.62 ยฑ31.73 / 467.88 ms โ 377.66 / 422.84 ยฑ32.42 / 471.79 ms โ no change โ
โ QQuery 56 โ 539.40 / 604.00 ยฑ40.03 / 657.80 ms โ 533.92 / 590.93 ยฑ40.14 / 643.35 ms โ no change โ
โ QQuery 57 โ 1081.57 / 1195.64 ยฑ77.51 / 1306.15 ms โ 1082.64 / 1193.48 ยฑ76.15 / 1303.68 ms โ no change โ
โ QQuery 58 โ 1162.77 / 1327.99 ยฑ94.05 / 1424.96 ms โ 1158.10 / 1328.50 ยฑ97.18 / 1421.19 ms โ no change โ
โ QQuery 59 โ 893.20 / 926.89 ยฑ51.48 / 1029.18 ms โ 908.78 / 935.70 ยฑ48.38 / 1032.42 ms โ no change โ
โ QQuery 60 โ 582.62 / 659.08 ยฑ68.20 / 745.11 ms โ 581.73 / 673.34 ยฑ64.62 / 749.09 ms โ no change โ
โ QQuery 61 โ 394.89 / 395.19 ยฑ0.22 / 395.48 ms โ 394.53 / 395.41 ยฑ0.77 / 396.27 ms โ no change โ
โ QQuery 62 โ 579.72 / 600.45 ยฑ24.67 / 631.03 ms โ 579.81 / 600.88 ยฑ25.18 / 632.36 ms โ no change โ
โ QQuery 63 โ 416.71 / 441.63 ยฑ28.37 / 478.44 ms โ 417.70 / 441.35 ยฑ27.63 / 475.64 ms โ no change โ
โ QQuery 64 โ 3126.25 / 3277.24 ยฑ103.91 / 3402.85 ms โ 3161.09 / 3287.78 ยฑ84.68 / 3387.49 ms โ no change โ
โ QQuery 65 โ 626.16 / 739.79 ยฑ67.30 / 826.18 ms โ 631.23 / 744.87 ยฑ64.98 / 826.51 ms โ no change โ
โ QQuery 66 โ 564.71 / 598.14 ยฑ41.98 / 677.42 ms โ 560.96 / 589.62 ยฑ43.97 / 677.29 ms โ no change โ
โ QQuery 67 โ 614.30 / 684.54 ยฑ45.68 / 737.55 ms โ 659.23 / 690.60 ยฑ27.67 / 723.51 ms โ no change โ
โ QQuery 68 โ 290.02 / 320.51 ยฑ20.76 / 354.71 ms โ 289.13 / 315.30 ยฑ15.80 / 336.05 ms โ no change โ
โ QQuery 69 โ 633.74 / 806.91 ยฑ94.48 / 907.86 ms โ 702.76 / 782.41 ยฑ84.04 / 904.29 ms โ no change โ
โ QQuery 70 โ 674.75 / 762.16 ยฑ66.28 / 866.95 ms โ 683.18 / 758.04 ยฑ58.78 / 853.09 ms โ no change โ
โ QQuery 71 โ 524.69 / 583.20 ยฑ45.71 / 664.16 ms โ 519.66 / 573.90 ยฑ54.06 / 670.26 ms โ no change โ
โ QQuery 72 โ 2997.80 / 3077.43 ยฑ67.68 / 3167.34 ms โ 3117.86 / 3221.68 ยฑ70.74 / 3322.51 ms โ no change โ
โ QQuery 73 โ 194.15 / 308.65 ยฑ60.24 / 362.54 ms โ 193.62 / 308.64 ยฑ62.37 / 360.69 ms โ no change โ
โ QQuery 74 โ 723.81 / 819.10 ยฑ56.80 / 892.66 ms โ 640.07 / 739.76 ยฑ75.21 / 837.79 ms โ +1.11x faster โ
โ QQuery 75 โ 922.46 / 1002.60 ยฑ50.56 / 1066.75 ms โ 921.85 / 981.19 ยฑ52.56 / 1042.41 ms โ no change โ
โ QQuery 76 โ 477.66 / 567.71 ยฑ68.38 / 652.90 ms โ 524.93 / 609.96 ยฑ63.72 / 686.89 ms โ 1.07x slower โ
โ QQuery 77 โ 653.75 / 701.70 ยฑ52.72 / 790.56 ms โ 604.85 / 696.04 ยฑ73.05 / 819.72 ms โ no change โ
โ QQuery 78 โ 707.72 / 856.09 ยฑ81.14 / 951.21 ms โ 715.09 / 851.98 ยฑ77.99 / 954.32 ms โ no change โ
โ QQuery 79 โ 448.87 / 510.46 ยฑ53.50 / 603.32 ms โ 456.37 / 507.24 ยฑ44.15 / 583.64 ms โ no change โ
โ QQuery 80 โ 651.00 / 715.34 ยฑ45.80 / 790.76 ms โ 658.42 / 722.90 ยฑ55.39 / 794.55 ms โ no change โ
โ QQuery 81 โ 476.65 / 618.13 ยฑ77.18 / 681.15 ms โ 477.79 / 613.40 ยฑ76.80 / 689.09 ms โ no change โ
โ QQuery 82 โ 557.52 / 644.04 ยฑ47.23 / 682.51 ms โ 557.14 / 643.67 ยฑ47.58 / 684.34 ms โ no change โ
โ QQuery 83 โ 862.98 / 952.57 ยฑ74.88 / 1028.36 ms โ 860.69 / 980.51 ยฑ105.48 / 1098.77 ms โ no change โ
โ QQuery 84 โ 442.23 / 502.69 ยฑ55.57 / 591.75 ms โ 439.42 / 502.05 ยฑ56.78 / 592.05 ms โ no change โ
โ QQuery 85 โ 820.09 / 888.19 ยฑ50.25 / 947.21 ms โ 816.18 / 887.07 ยฑ51.14 / 949.14 ms โ no change โ
โ QQuery 86 โ 247.98 / 315.19 ยฑ55.39 / 392.54 ms โ 249.19 / 315.39 ยฑ55.99 / 397.28 ms โ no change โ
โ QQuery 87 โ 612.93 / 682.64 ยฑ67.75 / 778.16 ms โ 609.20 / 673.65 ยฑ58.73 / 779.54 ms โ no change โ
โ QQuery 88 โ 738.12 / 739.39 ยฑ1.14 / 741.06 ms โ 737.47 / 738.34 ยฑ0.95 / 740.13 ms โ no change โ
โ QQuery 89 โ 427.17 / 459.36 ยฑ26.25 / 481.13 ms โ 426.22 / 461.05 ยฑ28.39 / 484.68 ms โ no change โ
โ QQuery 90 โ 427.64 / 467.66 ยฑ25.69 / 501.91 ms โ 425.25 / 466.28 ยฑ26.42 / 502.14 ms โ no change โ
โ QQuery 91 โ 525.00 / 664.53 ยฑ112.72 / 806.61 ms โ 523.68 / 663.32 ยฑ113.50 / 806.00 ms โ no change โ
โ QQuery 92 โ 373.67 / 446.73 ยฑ41.22 / 487.66 ms โ 371.49 / 439.06 ยฑ37.89 / 488.83 ms โ no change โ
โ QQuery 93 โ 467.68 / 484.21 ยฑ18.86 / 519.56 ms โ 466.79 / 481.71 ยฑ15.30 / 508.57 ms โ no change โ
โ QQuery 94 โ 479.73 / 530.13 ยฑ43.05 / 591.50 ms โ 480.42 / 530.73 ยฑ43.38 / 588.15 ms โ no change โ
โ QQuery 95 โ 692.97 / 777.00 ยฑ46.24 / 824.03 ms โ 694.33 / 778.60 ยฑ45.51 / 825.62 ms โ no change โ
โ QQuery 96 โ 403.44 / 446.46 ยฑ48.92 / 507.20 ms โ 403.23 / 445.88 ยฑ48.25 / 506.93 ms โ no change โ
โ QQuery 97 โ 453.60 / 518.38 ยฑ51.76 / 612.43 ms โ 508.30 / 535.67 ยฑ40.95 / 616.41 ms โ no change โ
โ QQuery 98 โ 394.58 / 440.31 ยฑ32.06 / 487.87 ms โ 401.63 / 442.58 ยฑ28.26 / 483.87 ms โ no change โ
โ QQuery 99 โ 508.26 / 620.00 ยฑ64.57 / 697.87 ms โ 507.78 / 611.66 ยฑ61.68 / 694.50 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 74914.28ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 75080.57ms โ
โ Average Time (HEAD) โ 756.71ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 758.39ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 96 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 1.1 GiB |
| Avg memory | 618.5 MiB |
| CPU user | 229.2s |
| CPU sys | 11.2s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 380.1s |
| Peak memory | 965.1 MiB |
| Avg memory | 540.9 MiB |
| CPU user | 241.1s |
| CPU sys | 11.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5204925346
> My intuition is to have an IO pipeline that uses memory, row counts and request concurrency to tune work, and feeds into CPU decoding. But itโs the decoding that determines _what_ data to fetch, either known or speculative (the row filter case) and that owns the data in flight (so it can do things like discard speculative data that proves non useful). The IO pipeline asks the decoder โhey iโm going to make my next 1-4MB request, what do you want in there?โ and the decoder + some IO policy decide what bytes they want.
Some other potentially related work is Lance's
https://github.com/lance-format/lance/blob/71c4aa2174971e98acb7e256fde1e1589024f5bc/rust/lance-encoding/src/decoder.rs#L424-L445
Here is their trait for the IO interface: https://github.com/lance-format/lance/blob/71c4aa2174971e98acb7e256fde1e1589024f5bc/rust/lance-encoding/src/lib.rs#L41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5197816153
Sorry I havenโt replied here. I hope to have time tomorrow to look in more detail.
Iโm not sure what the right architecture is yet, but my intuition is that:
- It should not be based on row groups or row numbers/batch sizes. Consider an int32 column vs a large string column: you can probably prefetch 1M rows of the int32 column each IO operation, you may only want a couple rows of the large string column.
- Like DuckDBs design it should be based on a decoupled async IO pipeline so we can make sure we keep the CPU and IO saturated
- It needs to account for vastly different IO profiles (SDD vs S3)
My intuition is to have an IO pipeline that uses memory, row counts and request concurrency to tune work, and feeds into CPU decoding. But itโs the decoding that determines *what* data to fetch, either known or speculative (the row filter case) and that owns the data in flight (so it can do things like discard speculative data that proves non useful). The IO pipeline asks the decoder โhey iโm going to make my next 1-4MB request, what do you want in there?โ and the decoder + some IO policy decide what bytes they want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951771659
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951762927-1014-vdhd9 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5297628109
> I have been thinking about this PR and how it could play into our "turn pushdown on by default" story
>
> We have put a lot of work in upstream to make applying filters during parquet decoding faster (thanks to @hhhizzz @haohuaijin and many others). However, enabling filter pushdown today **also** changes the I/O patterns (so that the all the needed columns are not fetched up front, but instead are fetched sequentially)
>
> I think one thing we could do with a nice I/O pre-fetching abstraction would be to start a request to fetch bytes for all columns required in a single request, so the I/O pattern is the same -- and then just enable the pushdown filter application once we already had the data ๐ค
I think that's "tricky" because the depends on the IO backend, no?
* For local SSD it can often be better to serialize the filters, so the first filter can reduce IO for the second, second for the third, etc.
* For object store backends it can be better to reduce the number of calls (for maybe slightly higher bandwidth usage). But bandwidth still important to consider (e.g. for AWS, S3 GET is only ~100MB/s, a standard high core instance might have only a couple of GB/s network bandwidth...)
I think for ClickBench it might not matter (no offset index?) - so perhaps we should special case this (just request the data).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951845762
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.26 / 4.09 ยฑ5.54 / 15.18 ms โ 1.21 / 3.95 ยฑ5.38 / 14.72 ms โ no change โ
โ QQuery 1 โ 1899.70 / 1928.55 ยฑ32.19 / 1989.64 ms โ 727.32 / 746.70 ยฑ19.51 / 779.84 ms โ +2.58x faster โ
โ QQuery 2 โ 2204.53 / 2273.36 ยฑ38.79 / 2319.14 ms โ 783.30 / 802.52 ยฑ13.48 / 824.53 ms โ +2.83x faster โ
โ QQuery 3 โ 2158.89 / 2188.58 ยฑ44.10 / 2275.07 ms โ 778.60 / 810.43 ยฑ18.59 / 833.55 ms โ +2.70x faster โ
โ QQuery 4 โ 2281.15 / 2342.49 ยฑ57.58 / 2427.21 ms โ 878.76 / 923.04 ยฑ24.05 / 941.99 ms โ +2.54x faster โ
โ QQuery 5 โ 2355.92 / 2422.78 ยฑ51.29 / 2514.29 ms โ 943.11 / 1015.14 ยฑ39.52 / 1053.13 ms โ +2.39x faster โ
โ QQuery 6 โ 1.27 / 1.41 ยฑ0.22 / 1.84 ms โ 1.26 / 1.42 ยฑ0.22 / 1.86 ms โ no change โ
โ QQuery 7 โ 1944.37 / 2010.23 ยฑ51.28 / 2102.32 ms โ 729.56 / 764.30 ยฑ29.57 / 812.02 ms โ +2.63x faster โ
โ QQuery 8 โ 2450.40 / 2477.87 ยฑ16.59 / 2497.54 ms โ 1023.30 / 1048.95 ยฑ16.22 / 1071.79 ms โ +2.36x faster โ
โ QQuery 9 โ 2640.44 / 2665.85 ยฑ25.37 / 2710.84 ms โ 1171.90 / 1226.77 ยฑ32.59 / 1263.43 ms โ +2.17x faster โ
โ QQuery 10 โ 2239.57 / 2309.75 ยฑ44.74 / 2374.34 ms โ 805.67 / 850.76 ยฑ23.52 / 871.67 ms โ +2.71x faster โ
โ QQuery 11 โ 2220.86 / 2275.82 ยฑ60.15 / 2384.89 ms โ 826.03 / 846.34 ยฑ14.87 / 868.25 ms โ +2.69x faster โ
โ QQuery 12 โ 2377.35 / 2426.38 ยฑ41.60 / 2496.82 ms โ 964.30 / 1006.74 ยฑ23.88 / 1035.05 ms โ +2.41x faster โ
โ QQuery 13 โ 2464.66 / 2479.72 ยฑ14.66 / 2507.36 ms โ 1090.73 / 1119.45 ยฑ21.85 / 1151.59 ms โ +2.22x faster โ
โ QQuery 14 โ 2396.46 / 2428.45 ยฑ17.05 / 2447.56 ms โ 995.99 / 1030.51 ยฑ20.34 / 1054.54 ms โ +2.36x faster โ
โ QQuery 15 โ 2384.87 / 2399.28 ยฑ15.16 / 2425.58 ms โ 947.36 / 977.71 ยฑ18.76 / 1002.11 ms โ +2.45x faster โ
โ QQuery 16 โ 2651.99 / 2689.71 ยฑ42.67 / 2759.98 ms โ 1291.91 / 1315.76 ยฑ17.13 / 1337.12 ms โ +2.04x faster โ
โ QQuery 17 โ 2627.83 / 2685.44 ยฑ34.42 / 2719.89 ms โ 1305.72 / 1327.67 ยฑ15.19 / 1341.59 ms โ +2.02x faster โ
โ QQuery 18 โ 3111.48 / 3158.40 ยฑ37.01 / 3212.31 ms โ 1862.76 / 1892.20 ยฑ30.64 / 1938.50 ms โ +1.67x faster โ
โ QQuery 19 โ 1745.03 / 1843.89 ยฑ72.05 / 1947.99 ms โ 744.10 / 760.88 ยฑ11.09 / 778.79 ms โ +2.42x faster โ
โ QQuery 20 โ 2654.25 / 2718.17 ยฑ66.88 / 2828.85 ms โ 1578.60 / 1644.03 ยฑ35.38 / 1680.86 ms โ +1.65x faster โ
โ QQuery 21 โ 2630.85 / 2707.82 ยฑ52.83 / 2768.85 ms โ 1799.08 / 1867.70 ยฑ63.52 / 1951.95 ms โ +1.45x faster โ
โ QQuery 22 โ 3087.79 / 3126.72 ยฑ26.75 / 3169.67 ms โ 2821.08 / 2885.40 ยฑ63.93 / 2989.55 ms โ +1.08x faster โ
โ QQuery 23 โ 4847.33 / 4973.43 ยฑ89.52 / 5090.72 ms โ 4978.92 / 5059.30 ยฑ59.61 / 5147.01 ms โ no change โ
โ QQuery 24 โ 451.32 / 501.07 ยฑ35.77 / 544.63 ms โ 331.96 / 366.13 ยฑ24.47 / 407.11 ms โ +1.37x faster โ
โ QQuery 25 โ 2349.38 / 2390.68 ยฑ27.54 / 2428.31 ms โ 877.12 / 904.10 ยฑ17.12 / 924.30 ms โ +2.64x faster โ
โ QQuery 26 โ 490.78 / 524.32 ยฑ27.69 / 557.09 ms โ 322.02 / 363.55 ยฑ27.92 / 407.38 ms โ +1.44x faster โ
โ QQuery 27 โ 2823.29 / 2902.44 ยฑ57.27 / 2970.89 ms โ 1746.98 / 1857.53 ยฑ93.08 / 2000.17 ms โ +1.56x faster โ
โ QQuery 28 โ 5097.80 / 5171.14 ยฑ41.48 / 5217.47 ms โ 4052.74 / 4203.06 ยฑ91.21 / 4296.42 ms โ +1.23x faster โ
โ QQuery 29 โ 2243.67 / 2286.67 ยฑ46.56 / 2363.37 ms โ 784.78 / 802.25 ยฑ12.88 / 816.47 ms โ +2.85x faster โ
โ QQuery 30 โ 2485.53 / 2581.20 ยฑ62.38 / 2649.36 ms โ 1004.64 / 1059.36 ยฑ36.28 / 1098.36 ms โ +2.44x faster โ
โ QQuery 31 โ 2401.05 / 2460.50 ยฑ52.83 / 2557.62 ms โ 1024.67 / 1054.40 ยฑ34.10 / 1114.16 ms โ +2.33x faster โ
โ QQuery 32 โ 2629.65 / 2677.64 ยฑ42.58 / 2741.76 ms โ 1188.31 / 1286.67 ยฑ110.57 / 1494.72 ms โ +2.08x faster โ
โ QQuery 33 โ 3459.14 / 3509.71 ยฑ44.86 / 3583.61 ms โ 2442.04 / 2460.03 ยฑ14.33 / 2485.70 ms โ +1.43x faster โ
โ QQuery 34 โ 3424.32 / 3516.37 ยฑ87.80 / 3659.75 ms โ 2432.51 / 2500.24 ยฑ50.42 / 2569.33 ms โ +1.41x faster โ
โ QQuery 35 โ 2392.22 / 2471.95 ยฑ54.06 / 2548.18 ms โ 948.08 / 977.51 ยฑ18.58 / 997.31 ms โ +2.53x faster โ
โ QQuery 36 โ 143.63 / 290.07 ยฑ89.56 / 403.94 ms โ 180.29 / 223.38 ยฑ29.19 / 270.53 ms โ +1.30x faster โ
โ QQuery 37 โ 135.20 / 251.61 ยฑ80.12 / 357.99 ms โ 86.98 / 139.51 ยฑ52.40 / 239.23 ms โ +1.80x faster โ
โ QQuery 38 โ 130.91 / 269.91 ยฑ85.68 / 376.13 ms โ 144.69 / 190.72 ยฑ38.78 / 234.58 ms โ +1.42x faster โ
โ QQuery 39 โ 195.96 / 327.52 ยฑ95.29 / 453.54 ms โ 256.46 / 336.27 ยฑ53.83 / 414.90 ms โ no change โ
โ QQuery 40 โ 115.94 / 239.26 ยฑ84.77 / 347.74 ms โ 119.71 / 152.82 ยฑ30.90 / 204.32 ms โ +1.57x faster โ
โ QQuery 41 โ 115.06 / 234.56 ยฑ83.74 / 345.18 ms โ 116.09 / 171.36 ยฑ41.77 / 237.72 ms โ +1.37x faster โ
โ QQuery 42 โ 113.43 / 240.32 ยฑ81.75 / 344.73 ms โ 82.09 / 124.47 ยฑ46.54 / 214.38 ms โ +1.93x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 89385.15ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 49101.06ms โ
โ Average Time (HEAD) โ 2078.72ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 1141.89ms โ
โ Queries Faster โ 39 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 4 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 450.1s |
| Peak memory | 9.2 GiB |
| Avg memory | 1.8 GiB |
| CPU user | 999.2s |
| CPU sys | 109.3s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 250.1s |
| Peak memory | 10.4 GiB |
| Avg memory | 2.3 GiB |
| CPU user | 995.2s |
| CPU sys | 92.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951208233
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951200496-1003-xqcct 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951206900
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951200496-1002-j44rs 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951436110
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951427817-1009-cr5t7 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951504907
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951427817)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.25 / 4.10 ยฑ5.57 / 15.24 ms โ 1.23 / 4.05 ยฑ5.50 / 15.06 ms โ no change โ
โ QQuery 1 โ 1885.45 / 1929.06 ยฑ41.60 / 2007.55 ms โ 728.11 / 746.96 ยฑ19.05 / 778.40 ms โ +2.58x faster โ
โ QQuery 2 โ 2160.87 / 2232.44 ยฑ54.64 / 2285.33 ms โ 782.88 / 801.80 ยฑ13.32 / 821.66 ms โ +2.78x faster โ
โ QQuery 3 โ 2151.58 / 2227.08 ยฑ47.31 / 2292.47 ms โ 777.39 / 810.45 ยฑ19.35 / 833.64 ms โ +2.75x faster โ
โ QQuery 4 โ 2345.35 / 2363.44 ยฑ22.41 / 2406.06 ms โ 891.99 / 922.88 ยฑ16.07 / 935.92 ms โ +2.56x faster โ
โ QQuery 5 โ 2358.82 / 2398.87 ยฑ29.03 / 2437.10 ms โ 946.88 / 1019.02 ยฑ40.32 / 1056.94 ms โ +2.35x faster โ
โ QQuery 6 โ 1.26 / 1.40 ยฑ0.22 / 1.84 ms โ 1.25 / 1.41 ยฑ0.21 / 1.83 ms โ no change โ
โ QQuery 7 โ 1908.66 / 1986.85 ยฑ57.91 / 2079.80 ms โ 731.80 / 766.37 ยฑ30.54 / 816.76 ms โ +2.59x faster โ
โ QQuery 8 โ 2404.68 / 2473.29 ยฑ58.82 / 2557.16 ms โ 998.26 / 1058.74 ยฑ33.99 / 1087.24 ms โ +2.34x faster โ
โ QQuery 9 โ 2651.86 / 2756.05 ยฑ65.72 / 2829.02 ms โ 1172.96 / 1230.49 ยฑ34.58 / 1271.37 ms โ +2.24x faster โ
โ QQuery 10 โ 2211.22 / 2270.46 ยฑ53.06 / 2368.39 ms โ 802.09 / 851.58 ยฑ25.53 / 873.93 ms โ +2.67x faster โ
โ QQuery 11 โ 2219.12 / 2297.36 ยฑ56.50 / 2358.53 ms โ 828.28 / 848.52 ยฑ15.92 / 872.29 ms โ +2.71x faster โ
โ QQuery 12 โ 2410.96 / 2453.10 ยฑ34.18 / 2499.73 ms โ 984.75 / 1010.82 ยฑ16.27 / 1034.15 ms โ +2.43x faster โ
โ QQuery 13 โ 2447.61 / 2487.79 ยฑ33.35 / 2539.14 ms โ 1131.03 / 1140.07 ยฑ6.50 / 1148.41 ms โ +2.18x faster โ
โ QQuery 14 โ 2409.09 / 2438.69 ยฑ27.81 / 2473.81 ms โ 1013.24 / 1030.28 ยฑ10.32 / 1045.66 ms โ +2.37x faster โ
โ QQuery 15 โ 2367.54 / 2426.65 ยฑ52.78 / 2514.29 ms โ 941.85 / 976.09 ยฑ26.80 / 1023.85 ms โ +2.49x faster โ
โ QQuery 16 โ 2614.23 / 2704.85 ยฑ69.70 / 2829.38 ms โ 1293.08 / 1309.61 ยฑ16.06 / 1335.14 ms โ +2.07x faster โ
โ QQuery 17 โ 2649.61 / 2704.22 ยฑ46.41 / 2788.36 ms โ 1330.93 / 1339.38 ยฑ8.27 / 1353.98 ms โ +2.02x faster โ
โ QQuery 18 โ 3162.40 / 3208.70 ยฑ32.99 / 3260.77 ms โ 1846.66 / 1907.69 ยฑ32.31 / 1933.62 ms โ +1.68x faster โ
โ QQuery 19 โ 1757.14 / 1833.47 ยฑ58.17 / 1932.49 ms โ 735.76 / 762.86 ยฑ16.76 / 781.71 ms โ +2.40x faster โ
โ QQuery 20 โ 2646.42 / 2711.44 ยฑ58.78 / 2797.96 ms โ 1612.82 / 1657.04 ยฑ47.64 / 1742.93 ms โ +1.64x faster โ
โ QQuery 21 โ 2683.30 / 2732.72 ยฑ32.84 / 2784.88 ms โ 1794.34 / 1886.24 ยฑ53.65 / 1960.83 ms โ +1.45x faster โ
โ QQuery 22 โ 3125.74 / 3151.86 ยฑ18.61 / 3170.79 ms โ 2812.73 / 2902.76 ยฑ59.74 / 2983.72 ms โ +1.09x faster โ
โ QQuery 23 โ 4946.09 / 5056.73 ยฑ72.63 / 5168.18 ms โ 4921.29 / 5017.94 ยฑ118.21 / 5235.22 ms โ no change โ
โ QQuery 24 โ 487.49 / 508.92 ยฑ23.92 / 541.81 ms โ 300.63 / 350.32 ยฑ32.65 / 401.14 ms โ +1.45x faster โ
โ QQuery 25 โ 2256.13 / 2349.35 ยฑ77.43 / 2474.04 ms โ 841.84 / 884.28 ยฑ32.17 / 926.18 ms โ +2.66x faster โ
โ QQuery 26 โ 468.86 / 522.48 ยฑ60.67 / 635.82 ms โ 319.28 / 354.91 ยฑ20.74 / 380.38 ms โ +1.47x faster โ
โ QQuery 27 โ 2736.42 / 2841.89 ยฑ63.70 / 2926.75 ms โ 1759.48 / 1805.63 ยฑ53.21 / 1908.44 ms โ +1.57x faster โ
โ QQuery 28 โ 5061.24 / 5158.44 ยฑ62.10 / 5217.45 ms โ 4083.56 / 4168.71 ยฑ74.39 / 4271.46 ms โ +1.24x faster โ
โ QQuery 29 โ 2110.07 / 2222.69 ยฑ59.98 / 2287.86 ms โ 771.02 / 827.21 ยฑ29.57 / 854.70 ms โ +2.69x faster โ
โ QQuery 30 โ 2444.41 / 2480.86 ยฑ35.41 / 2547.41 ms โ 1047.40 / 1093.91 ยฑ26.86 / 1115.31 ms โ +2.27x faster โ
โ QQuery 31 โ 2336.98 / 2417.41 ยฑ48.46 / 2489.87 ms โ 992.90 / 1039.70 ยฑ27.15 / 1070.41 ms โ +2.33x faster โ
โ QQuery 32 โ 2555.46 / 2624.33 ยฑ50.05 / 2694.92 ms โ 1147.51 / 1240.17 ยฑ59.53 / 1314.85 ms โ +2.12x faster โ
โ QQuery 33 โ 3432.63 / 3447.52 ยฑ9.56 / 3458.87 ms โ 2452.68 / 2497.96 ยฑ34.08 / 2540.05 ms โ +1.38x faster โ
โ QQuery 34 โ 3435.15 / 3482.94 ยฑ31.26 / 3521.14 ms โ 2480.25 / 2509.02 ยฑ19.83 / 2535.72 ms โ +1.39x faster โ
โ QQuery 35 โ 2395.62 / 2430.96 ยฑ35.98 / 2499.42 ms โ 948.92 / 1034.53 ยฑ46.85 / 1086.25 ms โ +2.35x faster โ
โ QQuery 36 โ 139.60 / 295.06 ยฑ90.18 / 398.56 ms โ 129.13 / 217.17 ยฑ60.47 / 288.94 ms โ +1.36x faster โ
โ QQuery 37 โ 134.98 / 252.58 ยฑ81.79 / 369.73 ms โ 121.08 / 164.34 ยฑ41.78 / 234.09 ms โ +1.54x faster โ
โ QQuery 38 โ 130.10 / 273.66 ยฑ90.29 / 392.73 ms โ 143.51 / 187.91 ยฑ42.97 / 247.31 ms โ +1.46x faster โ
โ QQuery 39 โ 192.64 / 341.74 ยฑ91.71 / 440.39 ms โ 270.19 / 343.21 ยฑ56.97 / 426.38 ms โ no change โ
โ QQuery 40 โ 117.85 / 248.10 ยฑ84.36 / 350.64 ms โ 114.04 / 147.16 ยฑ29.34 / 195.90 ms โ +1.69x faster โ
โ QQuery 41 โ 116.66 / 242.80 ยฑ81.00 / 344.59 ms โ 85.70 / 155.74 ยฑ49.67 / 215.60 ms โ +1.56x faster โ
โ QQuery 42 โ 113.88 / 240.39 ยฑ81.54 / 344.86 ms โ 113.67 / 153.26 ยฑ37.74 / 220.17 ms โ +1.57x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 89232.75ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 49178.19ms โ
โ Average Time (HEAD) โ 2075.18ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 1143.68ms โ
โ Queries Faster โ 39 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 4 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 450.1s |
| Peak memory | 9.3 GiB |
| Avg memory | 1.8 GiB |
| CPU user | 1011.3s |
| CPU sys | 114.3s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 250.1s |
| Peak memory | 10.3 GiB |
| Avg memory | 2.3 GiB |
| CPU user | 1001.7s |
| CPU sys | 96.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5197454577
> * Should we just configure a fixed budget or should we do something with bandwidth?
A classic way to model this would be with another "number of outstanding requests" limit -- which would limit how many outstanding requests could be going (which is a reasonable proxy for bandwidth). For users with large bandwidth / trying to maxxx out the execution, they could crank it up to 10 or 20 or something.
> * The interaction with filter pushdown is interesting - what is the best way to avoid serialization behind a large number of row filters while also minimizing bandwidth / decoding usage? The current implementation just plans the filters serially (`f1 -> f2 -> f3 -> f4 -> ... -> full projection`) without considering IO latency.
If we really wanted to get fancy, what I think we would do is
1. start fetching all the filter columns in parallel (subject to the outstanding request limit)
2. evaluate the filters as today serially
3. if we evaluates an earlier filter (say `f1` in your diagram) and pruned pages for a subsequent filter (say `f2`) we could start *another* racing read for the pruned pages in `f2` and if that finished before the original outstanding request we could cancel
Such a strategy seems somewhat complicated and on one end of the performance spectrum (maximum performance, at the cost of more requests and memory..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5175612108
> What can we expect for memory consumption? Will we use more memory because we prefetch?
Yes. The extra memory consumption is pretty low (and can be configured) - it is roughly max the IO budget x partitions (probably slightly higher because of overhead / Tokio tasks running, but generally much lower).
Thus 16 (partitions/cores) * 8 MiB = ~128 MiB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#discussion_r3724117458
##########
datafusion/common/src/config.rs:
##########
@@ -1133,6 +1133,13 @@ config_namespace! {
/// the hint, two reads will still be performed.
pub metadata_size_hint: Option<usize>, default = Some(512 * 1024)
+ /// (reading) If specified, the parquet reader will prefetch data for
+ /// subsequent row groups when the projected column chunks fit within
+ /// this many bytes. The required ranges for the current row group are
+ /// always read, even when they exceed this value. None disables data
+ /// prefetching.
+ pub prefetch_size: Option<usize>, default = None
Review Comment:
I agree this is an obvious and common strategy for reading from object store and we should add it
One thing I am wondering is if we can make this IO strategy more generic (for example, I can imagine a user wanting to support racing reads like the craziness described in https://github.com/apache/datafusion/pull/23492#issuecomment-5197454577
Where I am heading is "can we add an API / trait" that lets user customize the I/O behavior more?
For example, maybe the trait kind of like `madvise` that could have methods like
```rust
/// The reader will need the bytes in `range` at some point in the future
/// it will request the bytes in the same order as called by this function
/// though it may not read all of them
fn advise_bytes_needed(&self, range: Range<usize>)
```
And then we could include a default implementation that implemented memory limited prefetch ๐ค but users could provide their own implementations that did other things (like racing reads, etc)
##########
datafusion/datasource-parquet/src/push_decoder.rs:
##########
@@ -423,6 +446,68 @@ impl PushDecoderStreamState {
}
}
+/// Append projected column chunks from subsequent row groups to a decoder
+/// request while staying within `prefetch_size`.
+///
+/// The first entry in `rg_plan` is the row group responsible for `ranges`.
+/// Complete projected ranges for later row groups are added in scan order so
+/// the push decoder can stage them for future calls to `try_next_reader`.
+fn prefetch_row_group_ranges(
+ mut ranges: Vec<Range<u64>>,
+ buffered_bytes: u64,
+ prefetch_size: Option<usize>,
+ rg_plan: &VecDeque<RgPlanEntry>,
+ projection: &ProjectionMask,
+ metadata: &ParquetMetaData,
+ prefetched_row_groups: &mut HashSet<usize>,
+) -> Vec<Range<u64>> {
+ let Some(prefetch_size) = prefetch_size.filter(|size| *size > 0) else {
+ return ranges;
+ };
+
+ let requested_bytes = ranges
+ .iter()
+ .map(|range| range.end - range.start)
+ .sum::<u64>();
+ let mut staged_bytes = buffered_bytes.saturating_add(requested_bytes);
+ let budget = prefetch_size as u64;
+ if staged_bytes >= budget {
+ return ranges;
+ }
+
+ for entry in rg_plan.iter().skip(1) {
+ if prefetched_row_groups.contains(&entry.rg_index) {
+ continue;
+ }
+
+ let row_group = metadata.row_group(entry.rg_index);
Review Comment:
this would be a nice API to add upstream in the parquet crate probably
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5301788924
>
> * For local SSD it can often be better to serialize the filters, so the first filter can reduce IO for the second, second for the third, etc.
> * For object store backends it can be better to reduce the number of calls (for maybe slightly higher bandwidth usage). But bandwidth still important to consider (e.g. for AWS, S3 GET is only ~100MB/s, a standard high core instance might have only a couple of GB/s network bandwidth..., and other limits)
Yes for sure -- this is the key difference
Where I was heading is that previously, in my mind we can't turn on predicate pushdown because it caused slowdowns. The slowdowns were caused by two reasons:
1. It changed the I/O patterns (which slows down local SSD, but likely improves for object store)
2. It was computationally slower in some cases (figuring out bitmasks, rows that passed, etc)
I think after the great work from @hhhizzz @haohuaijin and others upstream in arrow-rs/parquet we are close (if not already) at the point where it is computationally faster.
All that remains is the different I/O. So if we can have a mode where we can turn on predicate pushdown and the I/O pattern is the same then we could (finally) turn on predicate pushdown by default.
> I think for ClickBench it might not matter (no offset index?) - so perhaps we should special case this (just request the columns).
This is a great point -- maybe that could be the trigger (if we are running with predicate pushdown but there is no page index, just fetch the column page data for all columns in one go) ๐ค
> Should we make a ClickBench dataset with a page index? Or an option on the CB benchmark if possible?
We could -- though I think we should figure out what exactly we are trying to measure / accomplish with such a benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5302213700
I filed a ticket to track the idea
- https://github.com/apache/datafusion/issues/24393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951439345
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and codex_parquet-pushdecoder-prefetch
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ codex_parquet-pushdecoder-prefetch โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.22 / 4.04 ยฑ5.52 / 15.08 ms โ 1.24 / 4.04 ยฑ5.52 / 15.08 ms โ no change โ
โ QQuery 1 โ 1850.13 / 1906.73 ยฑ39.71 / 1942.34 ms โ 1871.43 / 1908.95 ยฑ34.23 / 1962.31 ms โ no change โ
โ QQuery 2 โ 2181.87 / 2231.70 ยฑ36.25 / 2275.61 ms โ 2160.91 / 2256.26 ยฑ55.93 / 2321.58 ms โ no change โ
โ QQuery 3 โ 2200.30 / 2243.70 ยฑ37.61 / 2304.84 ms โ 2180.47 / 2255.62 ยฑ53.71 / 2336.48 ms โ no change โ
โ QQuery 4 โ 2302.75 / 2338.41 ยฑ28.09 / 2381.40 ms โ 2304.26 / 2339.10 ยฑ40.60 / 2417.98 ms โ no change โ
โ QQuery 5 โ 2366.29 / 2432.62 ยฑ37.94 / 2468.26 ms โ 2353.55 / 2430.84 ยฑ57.80 / 2500.36 ms โ no change โ
โ QQuery 6 โ 1.29 / 1.46 ยฑ0.22 / 1.88 ms โ 1.24 / 1.41 ยฑ0.22 / 1.84 ms โ no change โ
โ QQuery 7 โ 1947.48 / 2009.23 ยฑ48.64 / 2077.70 ms โ 1935.33 / 2003.83 ยฑ42.69 / 2069.49 ms โ no change โ
โ QQuery 8 โ 2402.04 / 2461.03 ยฑ41.53 / 2515.07 ms โ 2369.46 / 2456.67 ยฑ61.61 / 2558.57 ms โ no change โ
โ QQuery 9 โ 2632.67 / 2713.12 ยฑ45.42 / 2773.74 ms โ 2600.86 / 2678.22 ยฑ62.57 / 2771.92 ms โ no change โ
โ QQuery 10 โ 2171.85 / 2226.22 ยฑ35.68 / 2266.24 ms โ 2176.97 / 2259.60 ยฑ51.60 / 2334.85 ms โ no change โ
โ QQuery 11 โ 2216.31 / 2273.40 ยฑ42.90 / 2342.51 ms โ 2191.06 / 2263.43 ยฑ61.29 / 2345.52 ms โ no change โ
โ QQuery 12 โ 2358.58 / 2382.71 ยฑ23.70 / 2419.62 ms โ 2360.75 / 2413.09 ยฑ38.19 / 2479.04 ms โ no change โ
โ QQuery 13 โ 2466.81 / 2494.32 ยฑ32.82 / 2557.15 ms โ 2456.55 / 2508.92 ยฑ45.36 / 2572.74 ms โ no change โ
โ QQuery 14 โ 2412.50 / 2460.25 ยฑ40.27 / 2519.30 ms โ 2354.32 / 2449.55 ยฑ74.65 / 2583.23 ms โ no change โ
โ QQuery 15 โ 2393.91 / 2458.88 ยฑ57.37 / 2537.38 ms โ 2367.75 / 2440.54 ยฑ52.19 / 2525.78 ms โ no change โ
โ QQuery 16 โ 2672.42 / 2720.05 ยฑ43.52 / 2792.71 ms โ 2601.06 / 2699.86 ยฑ61.93 / 2788.09 ms โ no change โ
โ QQuery 17 โ 2654.74 / 2691.61 ยฑ45.49 / 2762.68 ms โ 2615.33 / 2682.51 ยฑ43.87 / 2722.70 ms โ no change โ
โ QQuery 18 โ 3163.70 / 3206.11 ยฑ24.49 / 3229.76 ms โ 3134.29 / 3205.02 ยฑ43.97 / 3268.78 ms โ no change โ
โ QQuery 19 โ 1732.29 / 1804.57 ยฑ60.29 / 1880.95 ms โ 1711.67 / 1790.40 ยฑ48.04 / 1850.39 ms โ no change โ
โ QQuery 20 โ 2693.09 / 2730.22 ยฑ38.66 / 2801.06 ms โ 2643.30 / 2682.11 ยฑ44.20 / 2760.92 ms โ no change โ
โ QQuery 21 โ 2623.81 / 2691.97 ยฑ52.40 / 2781.88 ms โ 2687.00 / 2754.70 ยฑ57.78 / 2848.72 ms โ no change โ
โ QQuery 22 โ 3118.41 / 3142.54 ยฑ14.34 / 3163.23 ms โ 3094.48 / 3113.84 ยฑ18.10 / 3147.80 ms โ no change โ
โ QQuery 23 โ 4955.66 / 5018.83 ยฑ81.16 / 5177.68 ms โ 4970.97 / 5065.50 ยฑ54.33 / 5135.58 ms โ no change โ
โ QQuery 24 โ 483.05 / 502.02 ยฑ19.54 / 538.05 ms โ 475.60 / 495.68 ยฑ15.29 / 519.11 ms โ no change โ
โ QQuery 25 โ 2248.61 / 2311.18 ยฑ42.26 / 2379.70 ms โ 2264.32 / 2328.16 ยฑ58.21 / 2435.85 ms โ no change โ
โ QQuery 26 โ 489.93 / 492.48 ยฑ2.26 / 496.37 ms โ 473.78 / 489.17 ยฑ10.09 / 505.10 ms โ no change โ
โ QQuery 27 โ 2757.09 / 2906.18 ยฑ85.85 / 2984.86 ms โ 2813.76 / 2885.73 ยฑ59.42 / 2963.15 ms โ no change โ
โ QQuery 28 โ 5187.18 / 5227.00 ยฑ46.07 / 5314.05 ms โ 5044.28 / 5168.71 ยฑ99.24 / 5338.87 ms โ no change โ
โ QQuery 29 โ 2132.72 / 2261.95 ยฑ86.23 / 2375.83 ms โ 2243.57 / 2306.91 ยฑ52.25 / 2397.97 ms โ no change โ
โ QQuery 30 โ 2404.23 / 2502.30 ยฑ58.36 / 2579.75 ms โ 2439.68 / 2538.51 ยฑ63.85 / 2640.20 ms โ no change โ
โ QQuery 31 โ 2354.43 / 2478.37 ยฑ86.12 / 2558.83 ms โ 2414.54 / 2448.75 ยฑ39.43 / 2525.07 ms โ no change โ
โ QQuery 32 โ 2467.66 / 2571.93 ยฑ85.93 / 2684.26 ms โ 2583.02 / 2598.67 ยฑ17.95 / 2632.57 ms โ no change โ
โ QQuery 33 โ 3431.96 / 3529.44 ยฑ62.87 / 3597.34 ms โ 3438.59 / 3481.09 ยฑ40.85 / 3552.07 ms โ no change โ
โ QQuery 34 โ 3419.88 / 3441.39 ยฑ13.62 / 3461.12 ms โ 3471.38 / 3519.06 ยฑ28.98 / 3549.82 ms โ no change โ
โ QQuery 35 โ 2365.07 / 2533.53 ยฑ93.18 / 2641.39 ms โ 2410.43 / 2451.28 ยฑ38.45 / 2522.30 ms โ no change โ
โ QQuery 36 โ 155.02 / 296.05 ยฑ84.29 / 391.08 ms โ 138.22 / 296.10 ยฑ93.90 / 401.95 ms โ no change โ
โ QQuery 37 โ 133.56 / 249.51 ยฑ78.51 / 357.71 ms โ 136.66 / 254.12 ยฑ81.74 / 372.59 ms โ no change โ
โ QQuery 38 โ 129.66 / 273.04 ยฑ90.43 / 392.82 ms โ 129.22 / 271.03 ยฑ89.60 / 389.52 ms โ no change โ
โ QQuery 39 โ 186.33 / 342.90 ยฑ94.42 / 451.01 ms โ 193.95 / 340.44 ยฑ92.97 / 448.80 ms โ no change โ
โ QQuery 40 โ 115.37 / 248.26 ยฑ85.99 / 353.32 ms โ 116.80 / 253.04 ยฑ85.82 / 359.23 ms โ no change โ
โ QQuery 41 โ 114.46 / 225.20 ยฑ73.94 / 344.53 ms โ 114.90 / 240.40 ยฑ81.38 / 345.96 ms โ 1.07x slower โ
โ QQuery 42 โ 115.49 / 241.78 ยฑ80.73 / 345.68 ms โ 114.79 / 225.62 ยฑ74.93 / 348.03 ms โ +1.07x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 89278.21ms โ
โ Total Time (codex_parquet-pushdecoder-prefetch) โ 89256.47ms โ
โ Average Time (HEAD) โ 2076.24ms โ
โ Average Time (codex_parquet-pushdecoder-prefetch) โ 2075.73ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 41 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 450.1s |
| Peak memory | 9.6 GiB |
| Avg memory | 1.8 GiB |
| CPU user | 1007.2s |
| CPU sys | 111.3s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 450.1s |
| Peak memory | 9.2 GiB |
| Avg memory | 1.8 GiB |
| CPU user | 1005.8s |
| CPU sys | 108.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951206438
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951200496)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951200496-1001-j9lxt 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951360661
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23492#issuecomment-4951351156)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4951351156-1005-9khmf 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing codex/parquet-pushdecoder-prefetch (4d02e1eda02a250a92194ccffc299ca5b4a6b5da) to 2880e10 (merge-base) [diff](https://github.com/apache/datafusion/compare/2880e1044c004fd2f4b4f82c9445812ad23f0411..4d02e1eda02a250a92194ccffc299ca5b4a6b5da) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Prefetch Parquet row groups with a bounded I/O budget [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #23492:
URL: https://github.com/apache/datafusion/pull/23492#issuecomment-4951762927
run benchmarks
```
env:
DATAFUSION_EXECUTION_PARQUET_PREFETCH_SIZE: "20971520"
SIMULATE_LATENCY: "true"
PUSHDOWN_FILTERS: "true"
REORDER_FILTERS: "true"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org