You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/04/21 16:05:14 UTC
[PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Dandandan opened a new pull request, #21766:
URL: https://github.com/apache/datafusion/pull/21766
## Which issue does this PR close?
Follow-up to #21351 (Dynamic work scheduling in FileStream), which closed #20529 and explicitly deferred *\"splitting files into smaller units (e.g. across row groups)\"* as future work. This PR implements that.
- Closes #.
## Rationale for this change
With #21351, sibling FileStreams already steal **whole files** from a `SharedWorkSource` queue. But a single large parquet file still bottlenecks on one worker — the other N−1 sibling partitions sit idle even though each row group is independently readable. This shows up on single-file queries (ClickBench-style) and on the long-tail large-file case in multi-file scans.
This PR adds row-group granularity: the worker that pops a file donates its other row groups back to the shared queue so idle siblings steal them.
## What changes are included in this PR?
**Donation path** (`datafusion/datasource-parquet/src/opener.rs`):
- New \`ParquetOpenState::SplitAndDonate\` state between \`LoadMetadata\` and \`PrepareFilters\`. After metadata load, the donor keeps the first eligible row group; each remaining one is pushed to the front of the shared queue as a \`PartitionedFile\` clone whose \`range\` is a one-byte \`FileRange\` at that row group's starting offset.
- The existing \`prune_by_range\` path matches that offset and scopes the stealer to exactly that row group — no new extension types, no metadata carried through \`PartitionedFile.extensions\`, no access-plan donation.
- If the caller pre-narrowed the scan with a \`file_range\` that still spans multiple row groups (byte-range file partitioning), splitting stays **inside** that range: donated ranges remain subsets of the caller's.
- Guards:
- Caller-supplied \`ParquetAccessPlan\` in \`extensions\` → respected as-is, no donation.
- Single row group in scope (whole file, or caller range isolating one RG) → no donation.
**Shared queue plumbing**:
- \`SharedWorkSource\` is now \`pub\`; gains \`push_front(items)\`, \`pop_front()\`, and \`Default\`.
- \`FileSource::create_morselizer\` takes an extra \`Option<SharedWorkSource>\` parameter so format-specific morselizers can participate in donation. Non-parquet sources ignore it.
- \`row_group_start_offset\` helper is extracted into \`row_group_filter.rs\` and reused by both \`prune_by_range\` and the new donation path.
**Trade-offs** (v1):
- Stealers re-read the parquet footer for their chunk. Object stores typically cache the range so this is cheap; carrying loaded metadata across siblings is left for a follow-up.
- If a sibling drains the shared queue *before* the donor has donated, that sibling terminates (it observes an empty queue at \`scan_state.rs\`'s \`ScanAndReturn::Done\`). Accepted for v1; fixing requires splitter-handles / queue wakeup and can be added separately.
## Are these changes tested?
Yes. Five new unit tests in \`datafusion/datasource-parquet/src/opener.rs\`:
- \`row_group_split_donates_remaining_row_groups\` — donor reads RG 0; three donated chunks each read exactly their row group, in file order.
- \`row_group_split_skips_single_row_group_file\` — no donation when the file has one row group.
- \`row_group_split_respects_caller_access_plan\` — \`ParquetAccessPlan\` in extensions suppresses donation; caller plan executes as specified.
- \`row_group_split_within_caller_file_range\` — caller byte range covering all RGs is split; donated ranges stay inside the caller range.
- \`row_group_split_skips_when_caller_range_covers_single_row_group\` — narrow caller range isolating one RG suppresses donation.
All existing \`datafusion-datasource\` and \`datafusion-datasource-parquet\` tests continue to pass. \`cargo clippy --all-targets --all-features -- -D warnings\` is clean on both crates.
## Are there any user-facing changes?
Performance only — faster single-file and tail-file scans under sibling work stealing. No semantic or API changes visible to SQL users. \`SharedWorkSource\` becomes \`pub\` (it was \`pub(crate)\`); \`FileSource::create_morselizer\` gains one parameter — default implementations ignore it.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290102619
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4290084954-1688-5l8cx 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (bfa1a938d62cfa452151d2439529e05e6882c6a5) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..bfa1a938d62cfa452151d2439529e05e6882c6a5) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291257140
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291026396-1704-txlv8 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291639928
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291527520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 0.72 / 1.08 ±0.65 / 2.37 ms │ 0.73 / 1.17 ±0.81 / 2.79 ms │ 1.08x slower │
│ QQuery 1 │ 14.42 / 14.89 ±0.69 / 16.26 ms │ 15.39 / 15.58 ±0.15 / 15.77 ms │ no change │
│ QQuery 2 │ 43.75 / 44.23 ±0.50 / 45.15 ms │ 38.39 / 38.70 ±0.22 / 39.06 ms │ +1.14x faster │
│ QQuery 3 │ 38.05 / 38.70 ±0.87 / 40.40 ms │ 33.04 / 33.75 ±1.10 / 35.93 ms │ +1.15x faster │
│ QQuery 4 │ 256.91 / 261.95 ±7.83 / 277.40 ms │ 250.93 / 256.12 ±5.05 / 264.30 ms │ no change │
│ QQuery 5 │ 435.96 / 438.15 ±1.99 / 440.85 ms │ 301.33 / 305.67 ±3.26 / 309.65 ms │ +1.43x faster │
│ QQuery 6 │ 6.26 / 6.54 ±0.22 / 6.85 ms │ 10.61 / 13.19 ±1.36 / 14.43 ms │ 2.02x slower │
│ QQuery 7 │ 16.08 / 16.36 ±0.24 / 16.78 ms │ 16.81 / 18.63 ±2.74 / 24.03 ms │ 1.14x slower │
│ QQuery 8 │ 363.01 / 366.96 ±3.46 / 373.05 ms │ 341.01 / 342.08 ±0.77 / 343.01 ms │ +1.07x faster │
│ QQuery 9 │ 620.04 / 630.41 ±6.47 / 638.10 ms │ 508.43 / 527.62 ±15.19 / 554.18 ms │ +1.19x faster │
│ QQuery 10 │ 90.92 / 93.82 ±4.81 / 103.42 ms │ 78.49 / 81.09 ±3.19 / 87.29 ms │ +1.16x faster │
│ QQuery 11 │ 105.26 / 106.87 ±2.69 / 112.23 ms │ 87.81 / 89.15 ±0.88 / 90.19 ms │ +1.20x faster │
│ QQuery 12 │ 442.18 / 450.49 ±6.61 / 462.45 ms │ 295.48 / 298.72 ±3.05 / 304.29 ms │ +1.51x faster │
│ QQuery 13 │ 494.57 / 508.98 ±13.16 / 533.00 ms │ 420.10 / 438.39 ±18.82 / 473.04 ms │ +1.16x faster │
│ QQuery 14 │ 440.96 / 452.90 ±7.62 / 461.76 ms │ 308.14 / 313.27 ±4.18 / 318.00 ms │ +1.45x faster │
│ QQuery 15 │ 304.66 / 313.70 ±11.50 / 336.27 ms │ 295.30 / 307.53 ±15.39 / 337.73 ms │ no change │
│ QQuery 16 │ 657.29 / 664.59 ±6.81 / 674.29 ms │ 661.41 / 677.12 ±22.31 / 720.39 ms │ no change │
│ QQuery 17 │ 647.30 / 663.80 ±9.72 / 675.36 ms │ 656.18 / 672.42 ±16.01 / 702.80 ms │ no change │
│ QQuery 18 │ 1289.71 / 1310.66 ±14.48 / 1328.58 ms │ 1289.91 / 1326.36 ±18.27 / 1336.97 ms │ no change │
│ QQuery 19 │ 35.65 / 35.91 ±0.19 / 36.23 ms │ 32.35 / 38.87 ±8.02 / 53.78 ms │ 1.08x slower │
│ QQuery 20 │ 625.68 / 640.31 ±10.83 / 656.44 ms │ 540.41 / 545.62 ±4.29 / 550.41 ms │ +1.17x faster │
│ QQuery 21 │ 711.75 / 729.21 ±10.44 / 740.87 ms │ 645.06 / 659.26 ±10.53 / 676.61 ms │ +1.11x faster │
│ QQuery 22 │ 1388.30 / 1404.99 ±14.96 / 1427.64 ms │ 1215.41 / 1227.65 ±13.35 / 1250.15 ms │ +1.14x faster │
│ QQuery 23 │ 3798.87 / 3841.65 ±29.92 / 3880.80 ms │ 3487.39 / 3496.48 ±10.38 / 3514.91 ms │ +1.10x faster │
│ QQuery 24 │ 215.90 / 220.13 ±4.92 / 228.67 ms │ 177.22 / 178.84 ±1.25 / 180.65 ms │ +1.23x faster │
│ QQuery 25 │ 183.19 / 184.93 ±1.54 / 187.19 ms │ 122.41 / 123.59 ±1.12 / 125.57 ms │ +1.50x faster │
│ QQuery 26 │ 217.43 / 221.85 ±4.09 / 229.42 ms │ 179.42 / 184.59 ±3.24 / 188.75 ms │ +1.20x faster │
│ QQuery 27 │ 762.86 / 772.27 ±7.43 / 784.94 ms │ 686.64 / 705.29 ±17.72 / 731.17 ms │ +1.09x faster │
│ QQuery 28 │ 3497.90 / 3508.74 ±14.54 / 3537.05 ms │ 2913.21 / 2926.33 ±8.17 / 2935.32 ms │ +1.20x faster │
│ QQuery 29 │ 46.97 / 63.93 ±33.06 / 130.04 ms │ 42.89 / 43.65 ±1.13 / 45.86 ms │ +1.46x faster │
│ QQuery 30 │ 416.36 / 427.62 ±17.97 / 463.41 ms │ 320.40 / 323.04 ±2.02 / 326.63 ms │ +1.32x faster │
│ QQuery 31 │ 390.73 / 413.76 ±29.30 / 470.75 ms │ 315.16 / 322.69 ±3.85 / 326.07 ms │ +1.28x faster │
│ QQuery 32 │ 1041.56 / 1055.89 ±13.25 / 1077.13 ms │ 1004.22 / 1010.63 ±3.70 / 1014.58 ms │ no change │
│ QQuery 33 │ 1454.88 / 1488.96 ±18.23 / 1505.04 ms │ 1483.30 / 1501.08 ±14.08 / 1519.31 ms │ no change │
│ QQuery 34 │ 1519.94 / 1563.22 ±36.70 / 1619.05 ms │ 1493.44 / 1524.85 ±30.56 / 1578.26 ms │ no change │
│ QQuery 35 │ 361.00 / 380.86 ±22.33 / 422.35 ms │ 290.64 / 302.81 ±10.05 / 321.08 ms │ +1.26x faster │
│ QQuery 36 │ 124.33 / 132.46 ±8.51 / 148.55 ms │ 76.61 / 82.52 ±4.99 / 88.68 ms │ +1.61x faster │
│ QQuery 37 │ 55.13 / 61.61 ±6.36 / 72.62 ms │ 46.46 / 53.01 ±7.76 / 66.35 ms │ +1.16x faster │
│ QQuery 38 │ 86.22 / 88.78 ±2.05 / 91.72 ms │ 48.91 / 52.16 ±3.74 / 59.26 ms │ +1.70x faster │
│ QQuery 39 │ 238.93 / 251.23 ±9.73 / 265.09 ms │ 155.86 / 160.21 ±3.76 / 166.07 ms │ +1.57x faster │
│ QQuery 40 │ 22.71 / 23.38 ±0.55 / 24.38 ms │ 18.11 / 19.61 ±2.24 / 24.05 ms │ +1.19x faster │
│ QQuery 41 │ 20.36 / 20.69 ±0.38 / 21.44 ms │ 17.01 / 17.18 ±0.10 / 17.31 ms │ +1.20x faster │
│ QQuery 42 │ 20.29 / 26.03 ±6.77 / 36.78 ms │ 16.61 / 18.80 ±3.99 / 26.77 ms │ +1.38x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 23943.47ms │
│ Total Time (row-group-morselization) │ 21275.32ms │
│ Average Time (HEAD) │ 556.82ms │
│ Average Time (row-group-morselization) │ 494.77ms │
│ Queries Faster │ 30 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 9 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_1 — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 125.0s |
| Peak memory | 30.7 GiB |
| Avg memory | 22.0 GiB |
| CPU user | 1142.2s |
| CPU sys | 67.7s |
| Peak spill | 0 B |
**clickbench_1 — branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 29.8 GiB |
| Avg memory | 22.4 GiB |
| CPU user | 1152.3s |
| CPU sys | 64.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4821356211
> @Dandandan just curious did you plan on pushing this forward?
Probably - at some point - but now my focus is elsewhere. Feel free to take it of course :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4835573366
So maybe we can find some way to get a performance improvement in prototype fashion and then harden/ productionize the approach 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290207442
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.19 / 4.62 ±6.77 / 18.15 ms │ 1.20 / 4.68 ±6.81 / 18.30 ms │ no change │
│ QQuery 1 │ 12.40 / 12.67 ±0.16 / 12.87 ms │ 18.41 / 19.56 ±0.73 / 20.37 ms │ 1.54x slower │
│ QQuery 2 │ 36.96 / 37.11 ±0.11 / 37.28 ms │ 40.36 / 40.80 ±0.53 / 41.82 ms │ 1.10x slower │
│ QQuery 3 │ 31.45 / 32.17 ±0.61 / 33.00 ms │ 35.54 / 35.87 ±0.26 / 36.19 ms │ 1.11x slower │
│ QQuery 4 │ 235.40 / 240.43 ±3.63 / 245.16 ms │ 250.00 / 253.32 ±2.06 / 255.95 ms │ 1.05x slower │
│ QQuery 5 │ 283.38 / 285.23 ±1.51 / 287.36 ms │ 288.36 / 295.93 ±12.38 / 320.38 ms │ no change │
│ QQuery 6 │ 6.30 / 6.94 ±0.39 / 7.26 ms │ 7.58 / 8.02 ±0.37 / 8.51 ms │ 1.16x slower │
│ QQuery 7 │ 13.71 / 14.34 ±1.12 / 16.57 ms │ 20.88 / 21.07 ±0.14 / 21.21 ms │ 1.47x slower │
│ QQuery 8 │ 330.39 / 332.81 ±2.56 / 337.04 ms │ 331.65 / 336.82 ±3.03 / 340.55 ms │ no change │
│ QQuery 9 │ 496.63 / 505.66 ±8.14 / 516.34 ms │ 492.30 / 507.25 ±12.96 / 527.13 ms │ no change │
│ QQuery 10 │ 74.15 / 75.20 ±0.83 / 76.49 ms │ 82.43 / 83.08 ±0.50 / 83.69 ms │ 1.10x slower │
│ QQuery 11 │ 84.15 / 85.15 ±0.56 / 85.88 ms │ 94.03 / 94.81 ±0.61 / 95.77 ms │ 1.11x slower │
│ QQuery 12 │ 275.33 / 282.52 ±8.00 / 296.95 ms │ 281.69 / 286.30 ±3.77 / 292.27 ms │ no change │
│ QQuery 13 │ 391.93 / 399.25 ±4.51 / 404.80 ms │ 406.88 / 415.09 ±6.80 / 422.76 ms │ no change │
│ QQuery 14 │ 287.43 / 292.52 ±6.06 / 301.85 ms │ 294.69 / 298.23 ±5.06 / 308.29 ms │ no change │
│ QQuery 15 │ 286.32 / 292.29 ±7.23 / 305.70 ms │ 286.22 / 295.33 ±6.06 / 302.75 ms │ no change │
│ QQuery 16 │ 622.92 / 631.80 ±7.15 / 641.03 ms │ 628.10 / 634.01 ±5.33 / 641.99 ms │ no change │
│ QQuery 17 │ 626.31 / 630.53 ±5.61 / 641.62 ms │ 631.64 / 640.27 ±8.19 / 655.28 ms │ no change │
│ QQuery 18 │ 1257.97 / 1271.44 ±11.62 / 1289.48 ms │ 1276.42 / 1287.28 ±12.63 / 1310.27 ms │ no change │
│ QQuery 19 │ 28.80 / 29.25 ±0.33 / 29.80 ms │ 35.94 / 39.69 ±6.80 / 53.28 ms │ 1.36x slower │
│ QQuery 20 │ 517.32 / 528.10 ±7.52 / 537.94 ms │ 499.87 / 502.54 ±2.86 / 506.37 ms │ no change │
│ QQuery 21 │ 594.74 / 600.33 ±4.15 / 606.11 ms │ 591.78 / 593.66 ±1.40 / 595.20 ms │ no change │
│ QQuery 22 │ 1073.71 / 1076.64 ±4.02 / 1084.51 ms │ 1038.58 / 1044.40 ±3.87 / 1050.39 ms │ no change │
│ QQuery 23 │ 3333.15 / 3348.66 ±13.72 / 3369.88 ms │ 3231.74 / 3244.06 ±13.72 / 3270.00 ms │ no change │
│ QQuery 24 │ 41.89 / 42.79 ±0.74 / 43.88 ms │ 43.89 / 48.10 ±4.67 / 54.46 ms │ 1.12x slower │
│ QQuery 25 │ 113.15 / 114.99 ±1.28 / 116.98 ms │ 121.91 / 123.10 ±0.88 / 124.20 ms │ 1.07x slower │
│ QQuery 26 │ 42.47 / 42.61 ±0.16 / 42.92 ms │ 43.68 / 44.64 ±0.72 / 45.49 ms │ no change │
│ QQuery 27 │ 668.92 / 680.20 ±8.65 / 695.47 ms │ 658.73 / 664.01 ±3.88 / 669.60 ms │ no change │
│ QQuery 28 │ 2996.21 / 3018.34 ±16.70 / 3043.73 ms │ 2880.66 / 2902.76 ±14.72 / 2925.28 ms │ no change │
│ QQuery 29 │ 42.61 / 46.02 ±5.58 / 57.10 ms │ 45.84 / 48.68 ±4.31 / 57.24 ms │ 1.06x slower │
│ QQuery 30 │ 311.60 / 314.90 ±2.57 / 319.15 ms │ 316.25 / 318.95 ±3.17 / 324.69 ms │ no change │
│ QQuery 31 │ 305.37 / 307.29 ±1.12 / 308.52 ms │ 315.15 / 322.37 ±4.55 / 328.85 ms │ no change │
│ QQuery 32 │ 1003.88 / 1008.46 ±3.32 / 1012.43 ms │ 1017.34 / 1027.96 ±9.01 / 1044.12 ms │ no change │
│ QQuery 33 │ 1430.13 / 1443.76 ±9.71 / 1458.59 ms │ 1443.41 / 1460.09 ±13.30 / 1479.94 ms │ no change │
│ QQuery 34 │ 1449.61 / 1476.48 ±17.47 / 1500.44 ms │ 1456.92 / 1472.37 ±8.93 / 1482.24 ms │ no change │
│ QQuery 35 │ 290.68 / 298.53 ±8.95 / 315.67 ms │ 291.79 / 308.31 ±15.87 / 332.14 ms │ no change │
│ QQuery 36 │ 63.77 / 68.30 ±4.80 / 76.52 ms │ 63.67 / 72.08 ±6.84 / 83.24 ms │ 1.06x slower │
│ QQuery 37 │ 35.79 / 36.27 ±0.64 / 37.46 ms │ 36.25 / 38.76 ±2.34 / 43.16 ms │ 1.07x slower │
│ QQuery 38 │ 41.17 / 44.44 ±4.10 / 52.39 ms │ 39.68 / 42.98 ±4.25 / 51.08 ms │ no change │
│ QQuery 39 │ 125.96 / 135.62 ±5.28 / 141.40 ms │ 128.97 / 136.74 ±4.64 / 141.83 ms │ no change │
│ QQuery 40 │ 14.00 / 16.35 ±3.02 / 22.29 ms │ 15.26 / 16.91 ±3.03 / 22.97 ms │ no change │
│ QQuery 41 │ 13.85 / 14.12 ±0.18 / 14.31 ms │ 14.68 / 15.29 ±0.83 / 16.94 ms │ 1.08x slower │
│ QQuery 42 │ 13.29 / 14.54 ±2.01 / 18.54 ms │ 14.23 / 14.63 ±0.31 / 15.08 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20139.69ms │
│ Total Time (row-group-morselization) │ 20060.82ms │
│ Average Time (HEAD) │ 468.36ms │
│ Average Time (row-group-morselization) │ 466.53ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 15 │
│ Queries with No Change │ 28 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.9 GiB |
| Avg memory | 23.2 GiB |
| CPU user | 1071.3s |
| CPU sys | 62.6s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.0 GiB |
| Avg memory | 23.0 GiB |
| CPU user | 1086.2s |
| CPU sys | 63.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291443884
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291424691-1712-s7zfm 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292287899
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294299393
run benchmark tpch-10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291708154
🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291661325)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291661325-1718-zndhl 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe)
BENCH_NAME=clcikbrnch_1
BENCH_COMMAND=cargo bench --features=parquet --bench clcikbrnch_1
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296852869
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.19 / 4.75 ±6.95 / 18.66 ms │ 1.17 / 4.68 ±6.85 / 18.38 ms │ no change │
│ QQuery 1 │ 12.95 / 13.33 ±0.21 / 13.57 ms │ 14.22 / 14.87 ±0.40 / 15.33 ms │ 1.12x slower │
│ QQuery 2 │ 38.19 / 38.48 ±0.28 / 38.96 ms │ 38.84 / 39.26 ±0.35 / 39.89 ms │ no change │
│ QQuery 3 │ 32.09 / 32.68 ±0.57 / 33.44 ms │ 33.74 / 34.43 ±0.70 / 35.28 ms │ 1.05x slower │
│ QQuery 4 │ 253.47 / 256.22 ±3.29 / 262.22 ms │ 250.66 / 256.17 ±3.23 / 260.06 ms │ no change │
│ QQuery 5 │ 292.18 / 294.25 ±1.40 / 296.05 ms │ 291.33 / 293.67 ±1.63 / 296.20 ms │ no change │
│ QQuery 6 │ 6.40 / 7.06 ±0.48 / 7.71 ms │ 7.08 / 7.61 ±0.32 / 8.00 ms │ 1.08x slower │
│ QQuery 7 │ 14.56 / 14.70 ±0.12 / 14.87 ms │ 16.34 / 16.49 ±0.14 / 16.69 ms │ 1.12x slower │
│ QQuery 8 │ 340.99 / 342.98 ±1.89 / 345.76 ms │ 333.99 / 337.92 ±3.28 / 342.59 ms │ no change │
│ QQuery 9 │ 522.05 / 529.69 ±6.49 / 538.62 ms │ 515.71 / 523.63 ±7.05 / 535.73 ms │ no change │
│ QQuery 10 │ 76.67 / 78.66 ±1.39 / 80.74 ms │ 76.91 / 79.89 ±3.68 / 87.10 ms │ no change │
│ QQuery 11 │ 87.38 / 88.88 ±0.99 / 89.85 ms │ 88.97 / 91.69 ±4.22 / 100.10 ms │ no change │
│ QQuery 12 │ 283.95 / 289.01 ±4.74 / 297.05 ms │ 283.14 / 286.53 ±4.16 / 294.27 ms │ no change │
│ QQuery 13 │ 408.96 / 425.43 ±17.81 / 456.28 ms │ 403.10 / 408.49 ±4.66 / 414.94 ms │ no change │
│ QQuery 14 │ 290.80 / 293.37 ±1.90 / 296.00 ms │ 294.98 / 299.03 ±3.18 / 302.89 ms │ no change │
│ QQuery 15 │ 286.88 / 292.34 ±3.43 / 296.61 ms │ 296.14 / 302.16 ±6.27 / 312.84 ms │ no change │
│ QQuery 16 │ 628.48 / 636.01 ±5.08 / 642.64 ms │ 640.70 / 643.16 ±2.05 / 646.52 ms │ no change │
│ QQuery 17 │ 631.81 / 636.53 ±5.08 / 645.61 ms │ 639.67 / 645.62 ±4.87 / 652.93 ms │ no change │
│ QQuery 18 │ 1269.23 / 1283.28 ±14.05 / 1309.47 ms │ 1292.37 / 1315.59 ±17.82 / 1337.16 ms │ no change │
│ QQuery 19 │ 29.19 / 30.92 ±1.34 / 32.76 ms │ 30.96 / 31.61 ±0.81 / 33.17 ms │ no change │
│ QQuery 20 │ 522.08 / 528.42 ±7.92 / 543.37 ms │ 503.21 / 507.69 ±4.82 / 516.56 ms │ no change │
│ QQuery 21 │ 601.36 / 602.92 ±1.23 / 604.31 ms │ 586.64 / 591.13 ±3.52 / 595.34 ms │ no change │
│ QQuery 22 │ 1068.17 / 1073.47 ±3.44 / 1078.34 ms │ 1044.75 / 1051.97 ±5.85 / 1061.18 ms │ no change │
│ QQuery 23 │ 3335.06 / 3363.60 ±20.42 / 3392.84 ms │ 3266.02 / 3288.56 ±12.34 / 3302.42 ms │ no change │
│ QQuery 24 │ 41.92 / 42.22 ±0.31 / 42.78 ms │ 46.21 / 48.56 ±2.82 / 54.01 ms │ 1.15x slower │
│ QQuery 25 │ 115.78 / 119.35 ±5.01 / 129.25 ms │ 114.76 / 118.24 ±4.27 / 126.61 ms │ no change │
│ QQuery 26 │ 42.64 / 44.63 ±1.49 / 46.19 ms │ 47.33 / 48.13 ±0.88 / 49.66 ms │ 1.08x slower │
│ QQuery 27 │ 667.24 / 672.99 ±3.22 / 676.71 ms │ 654.62 / 662.03 ±6.12 / 673.16 ms │ no change │
│ QQuery 28 │ 3028.30 / 3044.27 ±12.19 / 3064.17 ms │ 2864.62 / 2877.38 ±8.98 / 2890.09 ms │ +1.06x faster │
│ QQuery 29 │ 42.86 / 48.12 ±9.33 / 66.76 ms │ 45.10 / 46.30 ±1.40 / 49.02 ms │ no change │
│ QQuery 30 │ 312.42 / 316.94 ±4.43 / 325.14 ms │ 313.68 / 318.79 ±5.84 / 329.80 ms │ no change │
│ QQuery 31 │ 306.71 / 315.85 ±6.48 / 324.38 ms │ 305.62 / 315.62 ±6.95 / 323.49 ms │ no change │
│ QQuery 32 │ 1020.32 / 1036.81 ±19.98 / 1074.38 ms │ 1032.22 / 1061.03 ±35.67 / 1123.85 ms │ no change │
│ QQuery 33 │ 1469.18 / 1487.10 ±17.94 / 1514.02 ms │ 1466.22 / 1482.31 ±17.87 / 1511.25 ms │ no change │
│ QQuery 34 │ 1468.18 / 1487.12 ±13.51 / 1503.51 ms │ 1468.93 / 1546.67 ±78.33 / 1687.76 ms │ no change │
│ QQuery 35 │ 294.45 / 299.20 ±3.76 / 304.23 ms │ 299.49 / 327.04 ±35.71 / 395.97 ms │ 1.09x slower │
│ QQuery 36 │ 61.68 / 65.22 ±2.17 / 67.54 ms │ 62.18 / 65.50 ±2.98 / 70.32 ms │ no change │
│ QQuery 37 │ 35.68 / 37.76 ±3.47 / 44.69 ms │ 35.70 / 39.15 ±3.11 / 43.11 ms │ no change │
│ QQuery 38 │ 39.42 / 43.79 ±4.00 / 49.31 ms │ 39.17 / 42.63 ±2.90 / 47.81 ms │ no change │
│ QQuery 39 │ 126.01 / 136.73 ±5.41 / 140.68 ms │ 134.34 / 142.38 ±5.05 / 148.03 ms │ no change │
│ QQuery 40 │ 14.69 / 14.98 ±0.18 / 15.24 ms │ 14.36 / 20.30 ±8.29 / 36.16 ms │ 1.35x slower │
│ QQuery 41 │ 14.40 / 16.19 ±3.20 / 22.58 ms │ 13.71 / 15.85 ±3.62 / 23.08 ms │ no change │
│ QQuery 42 │ 13.79 / 16.09 ±2.45 / 19.58 ms │ 13.56 / 15.05 ±2.56 / 20.15 ms │ +1.07x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20402.34ms │
│ Total Time (row-group-morselization) │ 20264.78ms │
│ Average Time (HEAD) │ 474.47ms │
│ Average Time (row-group-morselization) │ 471.27ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 8 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.2 GiB |
| Avg memory | 23.0 GiB |
| CPU user | 1082.4s |
| CPU sys | 65.1s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.3 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1090.1s |
| CPU sys | 68.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297752534
run benchmark tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290102481
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4290084954-1687-mnsfh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (bfa1a938d62cfa452151d2439529e05e6882c6a5) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..bfa1a938d62cfa452151d2439529e05e6882c6a5) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291610766
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291595788-1717-bq8ss 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (88d6ed8bceb8ff74a8701d8290ce238e72ac666c) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..88d6ed8bceb8ff74a8701d8290ce238e72ac666c) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291531004
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.19 / 4.70 ±6.86 / 18.43 ms │ 1.18 / 4.61 ±6.74 / 18.08 ms │ no change │
│ QQuery 1 │ 12.87 / 13.57 ±0.42 / 14.03 ms │ 14.61 / 15.02 ±0.36 / 15.66 ms │ 1.11x slower │
│ QQuery 2 │ 39.40 / 40.03 ±0.74 / 41.43 ms │ 39.52 / 39.80 ±0.22 / 40.20 ms │ no change │
│ QQuery 3 │ 32.01 / 32.87 ±0.61 / 33.74 ms │ 33.92 / 34.37 ±0.26 / 34.65 ms │ no change │
│ QQuery 4 │ 249.03 / 254.40 ±4.54 / 260.01 ms │ 247.94 / 250.68 ±1.59 / 252.58 ms │ no change │
│ QQuery 5 │ 293.27 / 295.52 ±1.91 / 298.87 ms │ 288.73 / 290.50 ±1.45 / 292.82 ms │ no change │
│ QQuery 6 │ 6.95 / 7.37 ±0.51 / 8.33 ms │ 7.07 / 7.32 ±0.25 / 7.76 ms │ no change │
│ QQuery 7 │ 14.52 / 14.69 ±0.11 / 14.82 ms │ 16.11 / 16.28 ±0.12 / 16.43 ms │ 1.11x slower │
│ QQuery 8 │ 338.31 / 343.71 ±3.28 / 348.37 ms │ 333.76 / 337.29 ±2.76 / 342.26 ms │ no change │
│ QQuery 9 │ 513.42 / 527.29 ±12.70 / 545.23 ms │ 495.99 / 514.92 ±12.90 / 529.65 ms │ no change │
│ QQuery 10 │ 77.52 / 78.31 ±0.66 / 79.16 ms │ 77.05 / 78.65 ±0.84 / 79.38 ms │ no change │
│ QQuery 11 │ 88.51 / 89.53 ±0.84 / 90.76 ms │ 88.98 / 90.14 ±1.05 / 91.85 ms │ no change │
│ QQuery 12 │ 287.74 / 289.71 ±1.78 / 292.55 ms │ 277.54 / 286.14 ±7.33 / 298.02 ms │ no change │
│ QQuery 13 │ 406.51 / 410.16 ±2.66 / 413.08 ms │ 397.35 / 411.27 ±9.43 / 426.58 ms │ no change │
│ QQuery 14 │ 292.70 / 293.61 ±0.75 / 294.59 ms │ 289.85 / 296.09 ±6.09 / 307.41 ms │ no change │
│ QQuery 15 │ 292.63 / 297.27 ±5.04 / 306.76 ms │ 287.96 / 294.57 ±6.98 / 305.84 ms │ no change │
│ QQuery 16 │ 635.27 / 647.02 ±8.57 / 659.59 ms │ 629.89 / 635.19 ±5.31 / 644.14 ms │ no change │
│ QQuery 17 │ 629.99 / 637.73 ±6.01 / 647.27 ms │ 625.85 / 645.44 ±15.32 / 670.68 ms │ no change │
│ QQuery 18 │ 1261.37 / 1296.91 ±27.01 / 1342.44 ms │ 1272.40 / 1286.21 ±11.12 / 1303.31 ms │ no change │
│ QQuery 19 │ 29.63 / 30.01 ±0.42 / 30.55 ms │ 30.83 / 36.66 ±11.37 / 59.39 ms │ 1.22x slower │
│ QQuery 20 │ 528.49 / 538.29 ±9.68 / 555.01 ms │ 495.38 / 506.82 ±13.82 / 533.85 ms │ +1.06x faster │
│ QQuery 21 │ 597.09 / 604.86 ±7.28 / 618.44 ms │ 586.33 / 589.01 ±2.08 / 592.56 ms │ no change │
│ QQuery 22 │ 1058.84 / 1071.06 ±8.17 / 1080.85 ms │ 1035.56 / 1037.61 ±1.75 / 1040.02 ms │ no change │
│ QQuery 23 │ 3350.25 / 3374.36 ±16.54 / 3399.38 ms │ 3264.92 / 3277.53 ±15.66 / 3306.28 ms │ no change │
│ QQuery 24 │ 41.87 / 42.96 ±0.99 / 44.80 ms │ 47.15 / 50.14 ±5.56 / 61.25 ms │ 1.17x slower │
│ QQuery 25 │ 115.40 / 117.36 ±2.68 / 122.67 ms │ 116.74 / 118.07 ±1.34 / 120.55 ms │ no change │
│ QQuery 26 │ 42.73 / 43.34 ±0.55 / 44.35 ms │ 47.66 / 48.33 ±0.36 / 48.64 ms │ 1.12x slower │
│ QQuery 27 │ 677.91 / 682.89 ±5.85 / 693.36 ms │ 655.39 / 661.57 ±6.04 / 671.87 ms │ no change │
│ QQuery 28 │ 3048.33 / 3059.35 ±8.53 / 3073.28 ms │ 2883.59 / 2937.38 ±45.26 / 3002.72 ms │ no change │
│ QQuery 29 │ 43.23 / 49.95 ±7.60 / 62.78 ms │ 45.44 / 49.53 ±3.72 / 55.54 ms │ no change │
│ QQuery 30 │ 313.66 / 316.57 ±2.05 / 318.61 ms │ 317.61 / 322.41 ±5.01 / 329.14 ms │ no change │
│ QQuery 31 │ 311.91 / 316.69 ±4.34 / 322.08 ms │ 312.02 / 318.60 ±3.50 / 321.75 ms │ no change │
│ QQuery 32 │ 1031.13 / 1039.49 ±6.08 / 1047.29 ms │ 1037.71 / 1088.37 ±69.21 / 1225.39 ms │ no change │
│ QQuery 33 │ 1477.80 / 1488.11 ±11.08 / 1509.36 ms │ 1488.44 / 1502.24 ±11.77 / 1523.79 ms │ no change │
│ QQuery 34 │ 1471.61 / 1488.34 ±12.43 / 1504.22 ms │ 1495.37 / 1510.32 ±12.38 / 1527.43 ms │ no change │
│ QQuery 35 │ 295.71 / 305.39 ±8.11 / 316.71 ms │ 297.10 / 301.85 ±4.65 / 309.51 ms │ no change │
│ QQuery 36 │ 62.28 / 67.50 ±4.66 / 73.97 ms │ 62.83 / 65.21 ±1.60 / 67.02 ms │ no change │
│ QQuery 37 │ 35.92 / 37.24 ±0.77 / 38.07 ms │ 36.37 / 43.14 ±6.23 / 53.98 ms │ 1.16x slower │
│ QQuery 38 │ 41.18 / 44.14 ±3.22 / 49.81 ms │ 42.67 / 45.48 ±2.59 / 49.98 ms │ no change │
│ QQuery 39 │ 137.30 / 141.13 ±2.62 / 145.18 ms │ 132.29 / 135.60 ±3.65 / 142.45 ms │ no change │
│ QQuery 40 │ 15.22 / 15.67 ±0.35 / 16.24 ms │ 14.67 / 16.81 ±2.97 / 22.35 ms │ 1.07x slower │
│ QQuery 41 │ 14.19 / 14.36 ±0.18 / 14.71 ms │ 14.10 / 18.24 ±4.83 / 24.98 ms │ 1.27x slower │
│ QQuery 42 │ 14.13 / 15.79 ±2.79 / 21.37 ms │ 13.79 / 13.97 ±0.13 / 14.15 ms │ +1.13x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20479.23ms │
│ Total Time (row-group-morselization) │ 20229.40ms │
│ Average Time (HEAD) │ 476.26ms │
│ Average Time (row-group-morselization) │ 470.45ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 8 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 29.8 GiB |
| Avg memory | 22.9 GiB |
| CPU user | 1089.3s |
| CPU sys | 63.2s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.7 GiB |
| Avg memory | 23.4 GiB |
| CPU user | 1096.6s |
| CPU sys | 63.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291537003
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291527520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291527520-1713-sv7j2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) using: clickbench_1
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290207782
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.88 / 7.38 ±0.82 / 9.02 ms │ 6.81 / 7.33 ±0.85 / 9.02 ms │ no change │
│ QQuery 2 │ 146.18 / 146.83 ±0.54 / 147.78 ms │ 145.42 / 146.23 ±0.60 / 147.19 ms │ no change │
│ QQuery 3 │ 112.08 / 113.65 ±0.93 / 114.71 ms │ 113.61 / 114.39 ±0.60 / 115.11 ms │ no change │
│ QQuery 4 │ 1215.64 / 1217.92 ±2.39 / 1222.13 ms │ 1213.16 / 1218.38 ±2.95 / 1222.01 ms │ no change │
│ QQuery 5 │ 171.31 / 173.09 ±1.47 / 175.68 ms │ 172.45 / 173.76 ±0.93 / 174.94 ms │ no change │
│ QQuery 6 │ 805.27 / 824.20 ±14.36 / 839.62 ms │ 810.76 / 822.93 ±11.38 / 836.60 ms │ no change │
│ QQuery 7 │ 332.14 / 335.32 ±2.59 / 338.49 ms │ 334.89 / 336.74 ±1.21 / 338.73 ms │ no change │
│ QQuery 8 │ 111.33 / 112.45 ±1.33 / 115.04 ms │ 112.77 / 113.73 ±0.91 / 115.24 ms │ no change │
│ QQuery 9 │ 100.08 / 103.38 ±2.57 / 105.63 ms │ 101.21 / 104.95 ±1.89 / 106.24 ms │ no change │
│ QQuery 10 │ 101.11 / 101.58 ±0.48 / 102.29 ms │ 101.85 / 102.20 ±0.26 / 102.56 ms │ no change │
│ QQuery 11 │ 815.83 / 819.15 ±2.67 / 823.35 ms │ 807.74 / 820.27 ±6.39 / 824.91 ms │ no change │
│ QQuery 12 │ 42.69 / 43.14 ±0.45 / 43.90 ms │ 42.54 / 42.93 ±0.35 / 43.54 ms │ no change │
│ QQuery 13 │ 386.73 / 388.01 ±1.08 / 389.87 ms │ 385.45 / 389.55 ±4.05 / 397.31 ms │ no change │
│ QQuery 14 │ 974.11 / 982.89 ±5.39 / 989.63 ms │ 971.35 / 981.01 ±9.04 / 995.38 ms │ no change │
│ QQuery 15 │ 14.31 / 14.75 ±0.30 / 15.12 ms │ 14.28 / 14.60 ±0.31 / 15.16 ms │ no change │
│ QQuery 16 │ 7.22 / 7.29 ±0.10 / 7.49 ms │ 7.38 / 7.53 ±0.10 / 7.68 ms │ no change │
│ QQuery 17 │ 220.49 / 222.78 ±1.79 / 225.02 ms │ 220.38 / 221.72 ±0.87 / 222.89 ms │ no change │
│ QQuery 18 │ 121.26 / 122.28 ±0.99 / 124.01 ms │ 121.90 / 122.85 ±0.86 / 124.36 ms │ no change │
│ QQuery 19 │ 152.12 / 153.31 ±1.23 / 155.59 ms │ 152.51 / 154.68 ±2.11 / 158.11 ms │ no change │
│ QQuery 20 │ 12.86 / 13.26 ±0.43 / 14.08 ms │ 13.05 / 13.22 ±0.15 / 13.50 ms │ no change │
│ QQuery 21 │ 18.55 / 18.83 ±0.19 / 19.14 ms │ 18.78 / 19.05 ±0.30 / 19.51 ms │ no change │
│ QQuery 22 │ 474.99 / 478.91 ±2.79 / 483.61 ms │ 475.34 / 482.87 ±4.21 / 487.43 ms │ no change │
│ QQuery 23 │ 805.39 / 808.53 ±3.32 / 813.95 ms │ 810.11 / 813.78 ±2.76 / 818.66 ms │ no change │
│ QQuery 24 │ 370.65 / 372.49 ±3.00 / 378.44 ms │ 370.94 / 374.24 ±4.12 / 381.94 ms │ no change │
│ QQuery 25 │ 329.67 / 332.48 ±1.53 / 334.31 ms │ 330.79 / 332.77 ±1.97 / 336.28 ms │ no change │
│ QQuery 26 │ 75.94 / 76.49 ±0.34 / 76.92 ms │ 77.05 / 77.47 ±0.35 / 78.07 ms │ no change │
│ QQuery 27 │ 6.81 / 6.90 ±0.12 / 7.14 ms │ 7.01 / 7.65 ±1.14 / 9.93 ms │ 1.11x slower │
│ QQuery 28 │ 148.13 / 149.26 ±1.35 / 151.87 ms │ 146.93 / 148.47 ±1.67 / 151.57 ms │ no change │
│ QQuery 29 │ 271.62 / 274.40 ±2.96 / 279.94 ms │ 270.44 / 274.49 ±2.66 / 278.11 ms │ no change │
│ QQuery 30 │ 40.67 / 41.14 ±0.43 / 41.70 ms │ 41.04 / 41.50 ±0.48 / 42.32 ms │ no change │
│ QQuery 31 │ 162.44 / 164.67 ±3.32 / 171.06 ms │ 164.27 / 165.52 ±1.00 / 167.14 ms │ no change │
│ QQuery 32 │ 13.22 / 13.61 ±0.34 / 14.23 ms │ 13.08 / 13.47 ±0.22 / 13.70 ms │ no change │
│ QQuery 33 │ 137.47 / 138.50 ±0.59 / 139.21 ms │ 138.26 / 139.42 ±0.75 / 140.61 ms │ no change │
│ QQuery 34 │ 6.81 / 6.93 ±0.15 / 7.21 ms │ 6.89 / 7.10 ±0.20 / 7.47 ms │ no change │
│ QQuery 35 │ 99.13 / 100.14 ±0.87 / 101.19 ms │ 99.46 / 101.29 ±1.38 / 103.38 ms │ no change │
│ QQuery 36 │ 6.48 / 6.78 ±0.17 / 6.94 ms │ 6.67 / 7.13 ±0.29 / 7.57 ms │ 1.05x slower │
│ QQuery 37 │ 8.09 / 8.21 ±0.16 / 8.51 ms │ 8.22 / 8.40 ±0.21 / 8.80 ms │ no change │
│ QQuery 38 │ 84.61 / 85.19 ±0.49 / 85.99 ms │ 85.21 / 86.91 ±1.50 / 89.11 ms │ no change │
│ QQuery 39 │ 116.66 / 118.66 ±2.14 / 122.73 ms │ 118.19 / 120.31 ±2.38 / 124.67 ms │ no change │
│ QQuery 40 │ 102.63 / 105.13 ±3.33 / 111.57 ms │ 101.32 / 104.73 ±4.20 / 112.64 ms │ no change │
│ QQuery 41 │ 13.99 / 14.55 ±0.75 / 15.96 ms │ 14.11 / 15.14 ±1.67 / 18.46 ms │ no change │
│ QQuery 42 │ 105.50 / 107.16 ±1.71 / 110.27 ms │ 106.68 / 107.53 ±0.51 / 108.11 ms │ no change │
│ QQuery 43 │ 5.70 / 5.80 ±0.14 / 6.05 ms │ 5.79 / 5.88 ±0.11 / 6.09 ms │ no change │
│ QQuery 44 │ 11.39 / 11.60 ±0.15 / 11.82 ms │ 11.55 / 11.73 ±0.11 / 11.89 ms │ no change │
│ QQuery 45 │ 47.40 / 47.71 ±0.34 / 48.33 ms │ 47.59 / 48.05 ±0.34 / 48.59 ms │ no change │
│ QQuery 46 │ 8.25 / 8.38 ±0.11 / 8.57 ms │ 8.38 / 8.58 ±0.15 / 8.78 ms │ no change │
│ QQuery 47 │ 660.00 / 670.17 ±5.18 / 674.35 ms │ 666.15 / 673.72 ±4.39 / 677.54 ms │ no change │
│ QQuery 48 │ 271.77 / 273.61 ±1.20 / 275.14 ms │ 271.94 / 275.39 ±2.08 / 278.20 ms │ no change │
│ QQuery 49 │ 247.14 / 248.89 ±1.10 / 250.49 ms │ 248.40 / 250.03 ±1.25 / 251.61 ms │ no change │
│ QQuery 50 │ 200.23 / 205.58 ±3.86 / 211.76 ms │ 202.42 / 205.75 ±2.57 / 209.98 ms │ no change │
│ QQuery 51 │ 173.29 / 177.22 ±3.10 / 182.63 ms │ 177.75 / 180.80 ±2.25 / 184.61 ms │ no change │
│ QQuery 52 │ 105.32 / 107.38 ±2.40 / 112.02 ms │ 105.77 / 107.58 ±1.66 / 110.69 ms │ no change │
│ QQuery 53 │ 100.15 / 101.21 ±0.62 / 101.89 ms │ 100.98 / 102.17 ±1.15 / 104.07 ms │ no change │
│ QQuery 54 │ 141.02 / 142.30 ±0.70 / 143.11 ms │ 140.52 / 142.37 ±1.12 / 143.80 ms │ no change │
│ QQuery 55 │ 104.42 / 104.75 ±0.21 / 104.99 ms │ 104.80 / 106.45 ±1.29 / 108.65 ms │ no change │
│ QQuery 56 │ 136.95 / 137.93 ±0.61 / 138.60 ms │ 137.76 / 139.66 ±1.20 / 141.14 ms │ no change │
│ QQuery 57 │ 163.39 / 164.72 ±0.93 / 166.11 ms │ 164.00 / 166.13 ±1.51 / 167.93 ms │ no change │
│ QQuery 58 │ 306.50 / 308.15 ±1.36 / 309.92 ms │ 310.58 / 312.95 ±1.26 / 314.21 ms │ no change │
│ QQuery 59 │ 195.20 / 196.47 ±1.22 / 198.01 ms │ 196.35 / 197.34 ±1.33 / 199.95 ms │ no change │
│ QQuery 60 │ 137.82 / 140.24 ±1.96 / 143.12 ms │ 139.30 / 140.95 ±1.62 / 144.05 ms │ no change │
│ QQuery 61 │ 13.16 / 13.38 ±0.25 / 13.86 ms │ 13.41 / 13.56 ±0.16 / 13.88 ms │ no change │
│ QQuery 62 │ 849.66 / 856.51 ±5.31 / 865.52 ms │ 847.98 / 852.12 ±4.67 / 859.08 ms │ no change │
│ QQuery 63 │ 100.61 / 101.81 ±1.27 / 104.20 ms │ 100.77 / 101.55 ±0.74 / 102.59 ms │ no change │
│ QQuery 64 │ 651.92 / 656.94 ±4.29 / 663.28 ms │ 651.55 / 658.13 ±5.96 / 665.39 ms │ no change │
│ QQuery 65 │ 239.55 / 242.62 ±2.81 / 246.07 ms │ 242.19 / 244.42 ±2.96 / 250.18 ms │ no change │
│ QQuery 66 │ 213.29 / 222.54 ±11.11 / 237.95 ms │ 211.86 / 220.42 ±10.57 / 240.86 ms │ no change │
│ QQuery 67 │ 289.31 / 295.45 ±5.37 / 304.11 ms │ 292.04 / 295.91 ±6.91 / 309.73 ms │ no change │
│ QQuery 68 │ 8.54 / 8.69 ±0.18 / 9.03 ms │ 8.50 / 8.70 ±0.27 / 9.22 ms │ no change │
│ QQuery 69 │ 95.85 / 99.45 ±4.48 / 108.00 ms │ 96.60 / 97.79 ±1.30 / 100.25 ms │ no change │
│ QQuery 70 │ 303.63 / 311.47 ±6.01 / 320.98 ms │ 308.47 / 318.53 ±10.66 / 338.08 ms │ no change │
│ QQuery 71 │ 131.23 / 133.05 ±2.51 / 137.88 ms │ 133.29 / 135.61 ±2.31 / 139.65 ms │ no change │
│ QQuery 72 │ 583.55 / 594.33 ±6.81 / 604.74 ms │ 570.59 / 583.79 ±7.00 / 590.28 ms │ no change │
│ QQuery 73 │ 6.48 / 6.64 ±0.24 / 7.12 ms │ 6.54 / 6.73 ±0.23 / 7.17 ms │ no change │
│ QQuery 74 │ 515.18 / 520.75 ±2.83 / 522.70 ms │ 520.82 / 523.62 ±3.81 / 530.64 ms │ no change │
│ QQuery 75 │ 264.32 / 267.42 ±2.15 / 270.74 ms │ 266.84 / 268.84 ±1.73 / 272.00 ms │ no change │
│ QQuery 76 │ 128.87 / 129.75 ±1.04 / 131.36 ms │ 128.38 / 129.77 ±1.06 / 131.11 ms │ no change │
│ QQuery 77 │ 186.40 / 187.94 ±1.40 / 189.83 ms │ 186.53 / 187.12 ±0.56 / 187.83 ms │ no change │
│ QQuery 78 │ 331.83 / 332.80 ±0.86 / 334.26 ms │ 326.81 / 328.75 ±1.34 / 330.26 ms │ no change │
│ QQuery 79 │ 226.69 / 230.67 ±2.73 / 234.58 ms │ 225.24 / 228.79 ±3.53 / 235.42 ms │ no change │
│ QQuery 80 │ 321.55 / 322.93 ±1.41 / 325.51 ms │ 317.55 / 320.54 ±2.35 / 324.64 ms │ no change │
│ QQuery 81 │ 25.53 / 26.00 ±0.59 / 27.14 ms │ 25.23 / 25.54 ±0.27 / 26.02 ms │ no change │
│ QQuery 82 │ 38.52 / 39.49 ±0.71 / 40.61 ms │ 38.31 / 38.80 ±0.30 / 39.19 ms │ no change │
│ QQuery 83 │ 36.58 / 36.75 ±0.14 / 37.01 ms │ 35.98 / 36.33 ±0.24 / 36.61 ms │ no change │
│ QQuery 84 │ 45.88 / 46.11 ±0.19 / 46.43 ms │ 45.62 / 46.60 ±1.10 / 48.71 ms │ no change │
│ QQuery 85 │ 140.33 / 141.36 ±0.96 / 142.72 ms │ 139.58 / 139.82 ±0.24 / 140.24 ms │ no change │
│ QQuery 86 │ 36.30 / 36.70 ±0.27 / 37.07 ms │ 36.45 / 37.14 ±0.45 / 37.69 ms │ no change │
│ QQuery 87 │ 3.46 / 3.56 ±0.11 / 3.75 ms │ 3.47 / 3.55 ±0.10 / 3.73 ms │ no change │
│ QQuery 88 │ 98.77 / 101.37 ±2.48 / 104.79 ms │ 99.22 / 100.74 ±1.62 / 102.73 ms │ no change │
│ QQuery 89 │ 116.59 / 116.89 ±0.41 / 117.66 ms │ 115.25 / 116.57 ±1.79 / 120.11 ms │ no change │
│ QQuery 90 │ 22.09 / 22.87 ±1.08 / 24.97 ms │ 21.90 / 22.38 ±0.39 / 23.01 ms │ no change │
│ QQuery 91 │ 57.58 / 58.57 ±0.88 / 60.16 ms │ 57.52 / 58.05 ±0.44 / 58.59 ms │ no change │
│ QQuery 92 │ 55.62 / 56.15 ±0.46 / 56.96 ms │ 55.77 / 56.00 ±0.20 / 56.36 ms │ no change │
│ QQuery 93 │ 179.72 / 181.56 ±1.73 / 183.79 ms │ 179.29 / 180.32 ±1.20 / 182.60 ms │ no change │
│ QQuery 94 │ 59.38 / 59.84 ±0.35 / 60.44 ms │ 59.78 / 60.13 ±0.31 / 60.65 ms │ no change │
│ QQuery 95 │ 124.64 / 125.18 ±0.39 / 125.66 ms │ 124.38 / 125.01 ±0.72 / 126.09 ms │ no change │
│ QQuery 96 │ 67.18 / 68.19 ±0.89 / 69.73 ms │ 67.47 / 69.43 ±0.99 / 70.05 ms │ no change │
│ QQuery 97 │ 115.26 / 117.91 ±1.80 / 120.11 ms │ 116.91 / 117.81 ±0.77 / 119.15 ms │ no change │
│ QQuery 98 │ 148.12 / 148.76 ±0.56 / 149.48 ms │ 147.22 / 150.06 ±1.76 / 152.05 ms │ no change │
│ QQuery 99 │ 10683.84 / 10723.25 ±39.24 / 10798.41 ms │ 10745.61 / 10858.61 ±104.40 / 11008.80 ms │ no change │
└───────────┴──────────────────────────────────────────┴───────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 30029.32ms │
│ Total Time (row-group-morselization) │ 30201.38ms │
│ Average Time (HEAD) │ 303.33ms │
│ Average Time (row-group-morselization) │ 305.06ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 97 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.6 GiB |
| CPU user | 252.0s |
| CPU sys | 7.6s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.5 GiB |
| Avg memory | 5.6 GiB |
| CPU user | 252.7s |
| CPU sys | 8.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291348919
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.17 / 4.53 ±6.65 / 17.83 ms │ 1.19 / 4.53 ±6.62 / 17.78 ms │ no change │
│ QQuery 1 │ 12.88 / 13.31 ±0.22 / 13.48 ms │ 15.04 / 16.31 ±1.48 / 18.66 ms │ 1.22x slower │
│ QQuery 2 │ 37.48 / 37.75 ±0.24 / 38.14 ms │ 38.62 / 39.29 ±0.54 / 39.83 ms │ no change │
│ QQuery 3 │ 31.61 / 32.22 ±0.51 / 33.02 ms │ 33.58 / 34.28 ±0.67 / 35.50 ms │ 1.06x slower │
│ QQuery 4 │ 239.72 / 244.48 ±4.60 / 250.39 ms │ 243.70 / 246.58 ±2.18 / 249.48 ms │ no change │
│ QQuery 5 │ 283.71 / 285.12 ±1.03 / 286.85 ms │ 285.29 / 289.23 ±3.69 / 296.25 ms │ no change │
│ QQuery 6 │ 6.70 / 7.32 ±0.49 / 7.93 ms │ 6.76 / 7.27 ±0.27 / 7.54 ms │ no change │
│ QQuery 7 │ 14.02 / 14.24 ±0.19 / 14.58 ms │ 16.21 / 16.45 ±0.14 / 16.66 ms │ 1.16x slower │
│ QQuery 8 │ 329.12 / 331.26 ±1.83 / 333.85 ms │ 330.16 / 332.51 ±2.73 / 337.72 ms │ no change │
│ QQuery 9 │ 533.99 / 538.69 ±3.94 / 544.23 ms │ 482.72 / 524.52 ±22.38 / 545.84 ms │ no change │
│ QQuery 10 │ 74.73 / 75.79 ±1.41 / 78.55 ms │ 80.43 / 81.09 ±0.68 / 82.04 ms │ 1.07x slower │
│ QQuery 11 │ 87.39 / 89.38 ±1.67 / 92.45 ms │ 93.78 / 94.46 ±0.95 / 96.33 ms │ 1.06x slower │
│ QQuery 12 │ 281.35 / 286.51 ±4.12 / 291.16 ms │ 296.13 / 301.32 ±4.44 / 306.89 ms │ 1.05x slower │
│ QQuery 13 │ 397.34 / 407.22 ±9.22 / 422.36 ms │ 403.10 / 418.72 ±7.99 / 425.47 ms │ no change │
│ QQuery 14 │ 286.77 / 289.85 ±2.10 / 292.26 ms │ 289.70 / 297.26 ±5.31 / 305.82 ms │ no change │
│ QQuery 15 │ 282.13 / 289.43 ±5.77 / 297.42 ms │ 280.66 / 289.13 ±5.64 / 296.76 ms │ no change │
│ QQuery 16 │ 619.38 / 642.00 ±19.43 / 666.44 ms │ 628.17 / 631.52 ±4.62 / 640.68 ms │ no change │
│ QQuery 17 │ 639.73 / 661.37 ±11.07 / 670.98 ms │ 623.94 / 654.67 ±22.02 / 679.26 ms │ no change │
│ QQuery 18 │ 1263.19 / 1285.85 ±19.02 / 1316.35 ms │ 1269.63 / 1297.49 ±23.03 / 1335.62 ms │ no change │
│ QQuery 19 │ 29.01 / 30.45 ±2.31 / 34.98 ms │ 30.79 / 36.63 ±9.60 / 55.76 ms │ 1.20x slower │
│ QQuery 20 │ 521.20 / 524.62 ±4.33 / 533.05 ms │ 493.38 / 499.33 ±4.89 / 507.68 ms │ no change │
│ QQuery 21 │ 593.33 / 598.17 ±3.23 / 602.19 ms │ 597.85 / 603.50 ±9.21 / 621.83 ms │ no change │
│ QQuery 22 │ 1059.57 / 1092.14 ±16.88 / 1107.82 ms │ 1038.69 / 1045.26 ±7.75 / 1059.96 ms │ no change │
│ QQuery 23 │ 3361.32 / 3416.41 ±41.74 / 3489.55 ms │ 3228.38 / 3291.92 ±54.23 / 3371.34 ms │ no change │
│ QQuery 24 │ 42.29 / 42.99 ±0.53 / 43.44 ms │ 47.37 / 50.37 ±4.29 / 58.87 ms │ 1.17x slower │
│ QQuery 25 │ 116.78 / 119.86 ±3.39 / 126.32 ms │ 117.16 / 118.15 ±1.02 / 120.08 ms │ no change │
│ QQuery 26 │ 44.02 / 44.97 ±1.37 / 47.64 ms │ 47.65 / 50.26 ±4.56 / 59.35 ms │ 1.12x slower │
│ QQuery 27 │ 674.30 / 679.97 ±4.55 / 686.64 ms │ 661.70 / 668.53 ±4.05 / 672.61 ms │ no change │
│ QQuery 28 │ 3023.35 / 3082.51 ±33.21 / 3109.21 ms │ 2884.78 / 2916.44 ±27.82 / 2967.11 ms │ +1.06x faster │
│ QQuery 29 │ 42.82 / 45.48 ±3.98 / 53.35 ms │ 44.84 / 52.62 ±9.95 / 71.38 ms │ 1.16x slower │
│ QQuery 30 │ 332.14 / 336.85 ±3.47 / 341.46 ms │ 307.29 / 314.59 ±5.38 / 321.28 ms │ +1.07x faster │
│ QQuery 31 │ 324.74 / 332.03 ±8.72 / 348.32 ms │ 307.14 / 310.85 ±3.22 / 315.91 ms │ +1.07x faster │
│ QQuery 32 │ 1029.83 / 1053.38 ±19.89 / 1085.12 ms │ 1006.45 / 1013.49 ±5.73 / 1021.33 ms │ no change │
│ QQuery 33 │ 1440.14 / 1454.26 ±12.14 / 1475.18 ms │ 1453.04 / 1501.15 ±25.86 / 1526.86 ms │ no change │
│ QQuery 34 │ 1448.28 / 1533.01 ±44.76 / 1573.40 ms │ 1469.70 / 1505.32 ±37.21 / 1560.71 ms │ no change │
│ QQuery 35 │ 290.48 / 316.31 ±17.30 / 343.10 ms │ 285.88 / 294.59 ±5.84 / 301.17 ms │ +1.07x faster │
│ QQuery 36 │ 62.20 / 67.70 ±4.78 / 73.75 ms │ 61.83 / 70.72 ±6.25 / 77.46 ms │ no change │
│ QQuery 37 │ 36.22 / 43.07 ±5.87 / 50.22 ms │ 35.46 / 41.20 ±6.27 / 52.35 ms │ no change │
│ QQuery 38 │ 42.32 / 44.87 ±2.64 / 49.98 ms │ 40.63 / 42.79 ±1.99 / 46.55 ms │ no change │
│ QQuery 39 │ 124.48 / 134.43 ±7.64 / 147.38 ms │ 131.20 / 142.08 ±7.61 / 150.31 ms │ 1.06x slower │
│ QQuery 40 │ 14.62 / 15.39 ±1.14 / 17.61 ms │ 15.49 / 16.25 ±1.33 / 18.90 ms │ 1.06x slower │
│ QQuery 41 │ 13.88 / 15.12 ±1.80 / 18.63 ms │ 14.61 / 17.72 ±3.72 / 22.42 ms │ 1.17x slower │
│ QQuery 42 │ 13.40 / 13.67 ±0.18 / 13.88 ms │ 14.20 / 14.56 ±0.27 / 14.99 ms │ 1.06x slower │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20573.92ms │
│ Total Time (row-group-morselization) │ 20194.93ms │
│ Average Time (HEAD) │ 478.46ms │
│ Average Time (row-group-morselization) │ 469.65ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 14 │
│ Queries with No Change │ 25 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.2 GiB |
| Avg memory | 22.7 GiB |
| CPU user | 1090.3s |
| CPU sys | 66.6s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.4 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1090.3s |
| CPU sys | 66.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296852501
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 7.22 / 7.59 ±0.66 / 8.91 ms │ 7.02 / 7.45 ±0.78 / 9.01 ms │ no change │
│ QQuery 2 │ 147.99 / 148.41 ±0.31 / 148.95 ms │ 145.03 / 146.01 ±0.66 / 146.80 ms │ no change │
│ QQuery 3 │ 114.09 / 115.43 ±0.91 / 116.57 ms │ 114.22 / 115.40 ±1.27 / 117.57 ms │ no change │
│ QQuery 4 │ 1365.15 / 1374.92 ±8.90 / 1387.89 ms │ 1323.76 / 1339.25 ±10.92 / 1352.13 ms │ no change │
│ QQuery 5 │ 173.33 / 175.11 ±1.49 / 177.86 ms │ 172.35 / 173.52 ±0.70 / 174.27 ms │ no change │
│ QQuery 6 │ 849.63 / 859.99 ±12.96 / 885.52 ms │ 847.13 / 867.09 ±21.99 / 902.13 ms │ no change │
│ QQuery 7 │ 334.99 / 337.96 ±2.81 / 343.22 ms │ 338.09 / 339.27 ±1.25 / 341.62 ms │ no change │
│ QQuery 8 │ 114.47 / 115.35 ±0.82 / 116.50 ms │ 112.46 / 114.79 ±1.73 / 117.02 ms │ no change │
│ QQuery 9 │ 105.50 / 106.55 ±0.63 / 107.37 ms │ 100.89 / 102.67 ±1.81 / 105.90 ms │ no change │
│ QQuery 10 │ 102.77 / 103.41 ±0.46 / 104.10 ms │ 103.29 / 104.42 ±1.83 / 108.05 ms │ no change │
│ QQuery 11 │ 940.56 / 952.80 ±9.15 / 968.92 ms │ 925.76 / 938.14 ±9.45 / 952.45 ms │ no change │
│ QQuery 12 │ 44.03 / 44.20 ±0.13 / 44.39 ms │ 43.86 / 44.24 ±0.41 / 44.77 ms │ no change │
│ QQuery 13 │ 393.05 / 396.84 ±2.85 / 401.62 ms │ 392.29 / 393.54 ±1.26 / 395.57 ms │ no change │
│ QQuery 14 │ 977.85 / 989.17 ±8.87 / 1004.15 ms │ 974.36 / 985.43 ±7.47 / 996.14 ms │ no change │
│ QQuery 15 │ 15.05 / 15.33 ±0.24 / 15.78 ms │ 14.92 / 15.34 ±0.39 / 16.06 ms │ no change │
│ QQuery 16 │ 7.31 / 7.49 ±0.19 / 7.82 ms │ 7.39 / 7.50 ±0.13 / 7.75 ms │ no change │
│ QQuery 17 │ 222.27 / 223.46 ±1.48 / 226.31 ms │ 220.43 / 224.07 ±1.84 / 225.37 ms │ no change │
│ QQuery 18 │ 121.77 / 122.54 ±0.83 / 124.03 ms │ 122.61 / 123.71 ±1.29 / 126.24 ms │ no change │
│ QQuery 19 │ 153.26 / 154.16 ±0.59 / 154.87 ms │ 153.95 / 155.79 ±1.21 / 157.25 ms │ no change │
│ QQuery 20 │ 13.26 / 13.48 ±0.18 / 13.77 ms │ 13.25 / 13.49 ±0.20 / 13.81 ms │ no change │
│ QQuery 21 │ 19.43 / 19.65 ±0.19 / 19.90 ms │ 19.58 / 19.83 ±0.47 / 20.77 ms │ no change │
│ QQuery 22 │ 476.76 / 478.17 ±1.79 / 481.61 ms │ 485.21 / 488.71 ±3.15 / 493.89 ms │ no change │
│ QQuery 23 │ 830.16 / 835.55 ±3.47 / 840.09 ms │ 834.47 / 840.12 ±5.26 / 848.10 ms │ no change │
│ QQuery 24 │ 377.13 / 379.77 ±2.30 / 384.02 ms │ 378.20 / 381.52 ±2.71 / 384.79 ms │ no change │
│ QQuery 25 │ 335.19 / 338.21 ±2.71 / 342.43 ms │ 334.49 / 336.25 ±1.30 / 338.19 ms │ no change │
│ QQuery 26 │ 77.33 / 77.82 ±0.32 / 78.29 ms │ 77.14 / 77.80 ±0.39 / 78.33 ms │ no change │
│ QQuery 27 │ 6.80 / 6.99 ±0.16 / 7.28 ms │ 6.98 / 7.40 ±0.59 / 8.54 ms │ 1.06x slower │
│ QQuery 28 │ 148.60 / 149.43 ±0.57 / 150.34 ms │ 149.99 / 150.41 ±0.50 / 151.35 ms │ no change │
│ QQuery 29 │ 271.25 / 276.45 ±5.31 / 286.46 ms │ 273.81 / 275.14 ±0.83 / 276.15 ms │ no change │
│ QQuery 30 │ 41.57 / 42.40 ±0.74 / 43.53 ms │ 41.24 / 42.61 ±1.38 / 44.84 ms │ no change │
│ QQuery 31 │ 164.04 / 167.06 ±2.17 / 170.46 ms │ 165.56 / 167.17 ±1.15 / 168.55 ms │ no change │
│ QQuery 32 │ 13.72 / 13.99 ±0.19 / 14.22 ms │ 13.79 / 14.00 ±0.20 / 14.33 ms │ no change │
│ QQuery 33 │ 137.95 / 139.60 ±0.87 / 140.38 ms │ 138.92 / 140.61 ±1.16 / 141.87 ms │ no change │
│ QQuery 34 │ 6.97 / 7.10 ±0.11 / 7.29 ms │ 6.96 / 7.07 ±0.15 / 7.35 ms │ no change │
│ QQuery 35 │ 101.09 / 102.75 ±2.22 / 107.07 ms │ 100.71 / 101.70 ±0.72 / 102.85 ms │ no change │
│ QQuery 36 │ 6.55 / 6.95 ±0.23 / 7.23 ms │ 6.76 / 6.97 ±0.14 / 7.20 ms │ no change │
│ QQuery 37 │ 8.43 / 8.49 ±0.05 / 8.57 ms │ 8.26 / 8.35 ±0.07 / 8.44 ms │ no change │
│ QQuery 38 │ 86.09 / 86.99 ±0.89 / 88.56 ms │ 85.54 / 86.63 ±1.17 / 88.64 ms │ no change │
│ QQuery 39 │ 120.05 / 122.42 ±4.07 / 130.55 ms │ 118.00 / 120.89 ±3.38 / 127.38 ms │ no change │
│ QQuery 40 │ 104.74 / 109.35 ±2.68 / 112.29 ms │ 102.19 / 106.82 ±2.99 / 111.53 ms │ no change │
│ QQuery 41 │ 14.44 / 14.55 ±0.12 / 14.79 ms │ 14.21 / 15.65 ±2.52 / 20.69 ms │ 1.08x slower │
│ QQuery 42 │ 106.92 / 107.40 ±0.34 / 107.87 ms │ 107.01 / 107.55 ±0.43 / 108.10 ms │ no change │
│ QQuery 43 │ 5.78 / 5.93 ±0.12 / 6.14 ms │ 5.53 / 5.74 ±0.17 / 6.04 ms │ no change │
│ QQuery 44 │ 11.81 / 13.33 ±2.68 / 18.68 ms │ 11.58 / 11.74 ±0.09 / 11.81 ms │ +1.14x faster │
│ QQuery 45 │ 48.99 / 49.24 ±0.17 / 49.40 ms │ 48.21 / 49.03 ±0.47 / 49.57 ms │ no change │
│ QQuery 46 │ 8.70 / 9.37 ±1.07 / 11.50 ms │ 8.50 / 8.63 ±0.17 / 8.95 ms │ +1.09x faster │
│ QQuery 47 │ 725.84 / 735.91 ±7.71 / 745.94 ms │ 732.36 / 740.38 ±5.35 / 749.11 ms │ no change │
│ QQuery 48 │ 270.45 / 276.88 ±5.43 / 286.66 ms │ 275.99 / 279.53 ±3.55 / 284.59 ms │ no change │
│ QQuery 49 │ 247.81 / 249.47 ±1.25 / 251.36 ms │ 250.88 / 252.05 ±0.92 / 253.56 ms │ no change │
│ QQuery 50 │ 203.25 / 210.97 ±5.34 / 219.54 ms │ 204.85 / 210.44 ±4.32 / 217.31 ms │ no change │
│ QQuery 51 │ 176.21 / 180.23 ±3.33 / 185.58 ms │ 179.11 / 180.05 ±0.48 / 180.42 ms │ no change │
│ QQuery 52 │ 106.66 / 107.23 ±0.31 / 107.50 ms │ 107.29 / 108.67 ±1.64 / 111.85 ms │ no change │
│ QQuery 53 │ 101.76 / 103.50 ±1.98 / 107.31 ms │ 102.43 / 103.52 ±1.17 / 105.79 ms │ no change │
│ QQuery 54 │ 143.38 / 144.05 ±0.63 / 144.97 ms │ 144.41 / 145.78 ±0.78 / 146.81 ms │ no change │
│ QQuery 55 │ 105.58 / 106.44 ±0.53 / 107.23 ms │ 106.27 / 107.25 ±0.85 / 108.26 ms │ no change │
│ QQuery 56 │ 137.43 / 139.15 ±1.40 / 140.78 ms │ 139.92 / 141.13 ±1.06 / 142.56 ms │ no change │
│ QQuery 57 │ 164.90 / 167.23 ±2.41 / 171.87 ms │ 163.96 / 165.44 ±1.10 / 167.07 ms │ no change │
│ QQuery 58 │ 307.49 / 310.61 ±1.87 / 312.50 ms │ 310.72 / 311.88 ±0.97 / 313.17 ms │ no change │
│ QQuery 59 │ 196.32 / 198.12 ±1.86 / 201.54 ms │ 195.25 / 196.11 ±0.75 / 197.49 ms │ no change │
│ QQuery 60 │ 139.54 / 141.57 ±1.11 / 142.55 ms │ 140.96 / 142.25 ±1.31 / 144.49 ms │ no change │
│ QQuery 61 │ 13.47 / 13.59 ±0.10 / 13.72 ms │ 13.35 / 13.53 ±0.15 / 13.73 ms │ no change │
│ QQuery 62 │ 871.74 / 882.32 ±8.00 / 893.49 ms │ 875.37 / 891.50 ±9.18 / 902.61 ms │ no change │
│ QQuery 63 │ 101.73 / 102.92 ±1.58 / 106.05 ms │ 102.63 / 105.77 ±3.85 / 113.31 ms │ no change │
│ QQuery 64 │ 664.56 / 671.25 ±3.90 / 676.26 ms │ 667.23 / 673.00 ±4.63 / 680.60 ms │ no change │
│ QQuery 65 │ 246.69 / 248.78 ±1.93 / 251.14 ms │ 252.43 / 256.47 ±3.25 / 261.76 ms │ no change │
│ QQuery 66 │ 211.27 / 224.49 ±11.73 / 240.12 ms │ 221.29 / 231.23 ±12.42 / 252.32 ms │ no change │
│ QQuery 67 │ 301.30 / 307.40 ±9.07 / 325.37 ms │ 302.35 / 311.02 ±6.11 / 320.48 ms │ no change │
│ QQuery 68 │ 8.65 / 8.83 ±0.14 / 9.08 ms │ 8.63 / 8.82 ±0.20 / 9.20 ms │ no change │
│ QQuery 69 │ 97.77 / 100.58 ±3.31 / 106.65 ms │ 97.18 / 98.77 ±1.89 / 102.28 ms │ no change │
│ QQuery 70 │ 316.55 / 325.69 ±7.59 / 335.45 ms │ 319.01 / 327.92 ±8.74 / 343.39 ms │ no change │
│ QQuery 71 │ 131.99 / 136.68 ±4.42 / 142.40 ms │ 132.74 / 133.64 ±0.66 / 134.42 ms │ no change │
│ QQuery 72 │ 587.18 / 595.59 ±5.12 / 600.79 ms │ 598.11 / 602.99 ±3.30 / 607.18 ms │ no change │
│ QQuery 73 │ 6.62 / 6.76 ±0.17 / 7.09 ms │ 6.76 / 6.91 ±0.15 / 7.20 ms │ no change │
│ QQuery 74 │ 602.32 / 608.39 ±5.85 / 619.24 ms │ 609.78 / 616.03 ±4.24 / 621.76 ms │ no change │
│ QQuery 75 │ 267.38 / 269.38 ±2.53 / 274.32 ms │ 267.56 / 269.36 ±1.35 / 271.57 ms │ no change │
│ QQuery 76 │ 129.82 / 131.68 ±2.37 / 136.30 ms │ 131.42 / 133.15 ±0.98 / 134.45 ms │ no change │
│ QQuery 77 │ 186.06 / 187.84 ±2.02 / 191.77 ms │ 186.74 / 188.92 ±1.54 / 191.05 ms │ no change │
│ QQuery 78 │ 331.06 / 332.84 ±1.24 / 334.32 ms │ 333.28 / 336.50 ±1.85 / 338.97 ms │ no change │
│ QQuery 79 │ 235.24 / 238.48 ±3.24 / 244.43 ms │ 234.19 / 236.68 ±1.30 / 238.02 ms │ no change │
│ QQuery 80 │ 318.68 / 322.38 ±2.66 / 326.90 ms │ 319.97 / 320.48 ±0.49 / 321.29 ms │ no change │
│ QQuery 81 │ 25.68 / 26.30 ±0.42 / 26.94 ms │ 26.09 / 27.28 ±1.79 / 30.83 ms │ no change │
│ QQuery 82 │ 39.66 / 39.98 ±0.25 / 40.43 ms │ 39.79 / 40.29 ±0.43 / 40.86 ms │ no change │
│ QQuery 83 │ 37.36 / 37.84 ±0.53 / 38.79 ms │ 37.15 / 37.44 ±0.45 / 38.31 ms │ no change │
│ QQuery 84 │ 46.21 / 46.83 ±0.65 / 47.97 ms │ 46.61 / 46.77 ±0.16 / 47.00 ms │ no change │
│ QQuery 85 │ 139.95 / 141.42 ±2.11 / 145.61 ms │ 141.13 / 142.25 ±0.63 / 142.84 ms │ no change │
│ QQuery 86 │ 37.67 / 38.03 ±0.36 / 38.61 ms │ 37.63 / 37.89 ±0.20 / 38.13 ms │ no change │
│ QQuery 87 │ 3.43 / 3.54 ±0.17 / 3.87 ms │ 3.50 / 3.62 ±0.18 / 3.98 ms │ no change │
│ QQuery 88 │ 99.12 / 99.46 ±0.37 / 99.99 ms │ 99.92 / 102.23 ±2.52 / 106.96 ms │ no change │
│ QQuery 89 │ 116.41 / 118.83 ±3.27 / 125.29 ms │ 117.22 / 118.02 ±0.56 / 118.81 ms │ no change │
│ QQuery 90 │ 22.35 / 23.07 ±0.41 / 23.49 ms │ 22.49 / 22.84 ±0.26 / 23.10 ms │ no change │
│ QQuery 91 │ 57.84 / 58.92 ±0.68 / 59.93 ms │ 58.80 / 59.93 ±0.94 / 61.54 ms │ no change │
│ QQuery 92 │ 56.91 / 57.65 ±1.10 / 59.80 ms │ 57.08 / 57.72 ±0.35 / 58.10 ms │ no change │
│ QQuery 93 │ 180.95 / 182.86 ±1.54 / 185.48 ms │ 181.86 / 185.13 ±1.84 / 187.16 ms │ no change │
│ QQuery 94 │ 60.58 / 61.52 ±0.55 / 62.23 ms │ 60.70 / 61.25 ±0.38 / 61.90 ms │ no change │
│ QQuery 95 │ 125.68 / 127.41 ±1.42 / 130.00 ms │ 127.10 / 127.98 ±0.90 / 129.69 ms │ no change │
│ QQuery 96 │ 68.32 / 69.44 ±0.92 / 70.63 ms │ 67.46 / 69.36 ±1.09 / 70.43 ms │ no change │
│ QQuery 97 │ 117.24 / 118.69 ±1.18 / 119.96 ms │ 117.71 / 119.17 ±1.09 / 120.77 ms │ no change │
│ QQuery 98 │ 151.77 / 153.97 ±1.57 / 156.25 ms │ 150.95 / 152.14 ±1.39 / 154.77 ms │ no change │
│ QQuery 99 │ 10766.52 / 10937.58 ±134.68 / 11143.05 ms │ 10848.51 / 10878.73 ±25.54 / 10909.14 ms │ no change │
└───────────┴───────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 30963.75ms │
│ Total Time (row-group-morselization) │ 30926.69ms │
│ Average Time (HEAD) │ 312.77ms │
│ Average Time (row-group-morselization) │ 312.39ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 95 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.3 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 258.2s |
| CPU sys | 8.6s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.8 GiB |
| Avg memory | 5.7 GiB |
| CPU user | 260.0s |
| CPU sys | 8.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291372895
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.20 / 4.54 ±6.64 / 17.82 ms │ 1.16 / 4.52 ±6.65 / 17.82 ms │ no change │
│ QQuery 1 │ 13.23 / 13.53 ±0.29 / 14.05 ms │ 14.74 / 15.01 ±0.21 / 15.23 ms │ 1.11x slower │
│ QQuery 2 │ 38.36 / 38.77 ±0.31 / 39.23 ms │ 39.43 / 40.04 ±0.63 / 41.02 ms │ no change │
│ QQuery 3 │ 32.82 / 33.51 ±0.70 / 34.73 ms │ 35.74 / 36.37 ±0.33 / 36.70 ms │ 1.09x slower │
│ QQuery 4 │ 239.46 / 257.84 ±11.20 / 273.98 ms │ 240.90 / 256.68 ±13.47 / 276.98 ms │ no change │
│ QQuery 5 │ 285.70 / 298.07 ±6.73 / 305.99 ms │ 286.48 / 298.31 ±9.43 / 309.14 ms │ no change │
│ QQuery 6 │ 6.32 / 7.53 ±0.75 / 8.52 ms │ 6.86 / 7.90 ±0.68 / 8.86 ms │ no change │
│ QQuery 7 │ 14.57 / 14.75 ±0.14 / 15.00 ms │ 17.85 / 18.03 ±0.20 / 18.35 ms │ 1.22x slower │
│ QQuery 8 │ 330.39 / 340.82 ±8.96 / 353.99 ms │ 328.72 / 338.04 ±13.19 / 363.63 ms │ no change │
│ QQuery 9 │ 495.82 / 523.43 ±24.11 / 562.37 ms │ 510.51 / 525.84 ±13.51 / 546.44 ms │ no change │
│ QQuery 10 │ 74.59 / 75.67 ±1.17 / 77.82 ms │ 76.93 / 77.37 ±0.67 / 78.70 ms │ no change │
│ QQuery 11 │ 85.77 / 87.15 ±0.98 / 88.27 ms │ 92.63 / 93.63 ±0.60 / 94.17 ms │ 1.07x slower │
│ QQuery 12 │ 286.81 / 297.30 ±8.39 / 308.05 ms │ 275.98 / 288.78 ±10.47 / 304.75 ms │ no change │
│ QQuery 13 │ 432.82 / 440.13 ±6.25 / 449.96 ms │ 412.01 / 418.79 ±5.29 / 428.08 ms │ no change │
│ QQuery 14 │ 307.11 / 311.45 ±4.86 / 319.61 ms │ 297.26 / 305.27 ±5.65 / 312.04 ms │ no change │
│ QQuery 15 │ 311.21 / 315.12 ±3.96 / 322.38 ms │ 289.02 / 302.00 ±17.03 / 334.64 ms │ no change │
│ QQuery 16 │ 650.76 / 669.58 ±13.01 / 690.36 ms │ 645.45 / 657.77 ±7.75 / 665.55 ms │ no change │
│ QQuery 17 │ 638.97 / 657.88 ±11.83 / 671.67 ms │ 632.16 / 642.12 ±14.72 / 670.39 ms │ no change │
│ QQuery 18 │ 1295.60 / 1316.40 ±13.43 / 1336.78 ms │ 1302.97 / 1359.37 ±34.50 / 1409.67 ms │ no change │
│ QQuery 19 │ 29.10 / 30.75 ±2.68 / 36.04 ms │ 32.30 / 47.62 ±18.83 / 78.25 ms │ 1.55x slower │
│ QQuery 20 │ 523.92 / 534.95 ±12.39 / 558.15 ms │ 503.89 / 511.81 ±6.62 / 522.76 ms │ no change │
│ QQuery 21 │ 607.47 / 612.95 ±3.98 / 617.50 ms │ 596.12 / 603.03 ±3.92 / 607.75 ms │ no change │
│ QQuery 22 │ 1081.86 / 1091.35 ±10.96 / 1110.96 ms │ 1042.41 / 1051.65 ±8.56 / 1067.20 ms │ no change │
│ QQuery 23 │ 3381.18 / 3404.63 ±18.00 / 3425.77 ms │ 3278.51 / 3336.84 ±52.82 / 3412.53 ms │ no change │
│ QQuery 24 │ 42.06 / 50.86 ±15.29 / 81.41 ms │ 46.26 / 47.61 ±1.01 / 49.34 ms │ +1.07x faster │
│ QQuery 25 │ 114.08 / 119.58 ±5.16 / 128.63 ms │ 116.40 / 124.75 ±7.08 / 136.88 ms │ no change │
│ QQuery 26 │ 42.67 / 44.24 ±1.03 / 45.41 ms │ 46.87 / 48.92 ±3.06 / 54.99 ms │ 1.11x slower │
│ QQuery 27 │ 667.82 / 679.71 ±7.17 / 689.22 ms │ 662.79 / 675.39 ±7.78 / 682.62 ms │ no change │
│ QQuery 28 │ 3038.66 / 3094.52 ±35.63 / 3136.19 ms │ 2897.70 / 2911.16 ±9.19 / 2922.06 ms │ +1.06x faster │
│ QQuery 29 │ 44.47 / 48.01 ±4.42 / 56.35 ms │ 47.92 / 51.44 ±5.32 / 61.92 ms │ 1.07x slower │
│ QQuery 30 │ 312.54 / 327.85 ±12.58 / 342.48 ms │ 313.26 / 324.56 ±7.38 / 333.77 ms │ no change │
│ QQuery 31 │ 306.90 / 323.35 ±10.79 / 340.15 ms │ 320.76 / 328.00 ±6.12 / 335.48 ms │ no change │
│ QQuery 32 │ 1036.43 / 1059.40 ±21.58 / 1098.12 ms │ 1020.10 / 1052.81 ±26.36 / 1097.36 ms │ no change │
│ QQuery 33 │ 1450.58 / 1503.71 ±36.86 / 1553.76 ms │ 1568.80 / 1599.31 ±21.82 / 1631.94 ms │ 1.06x slower │
│ QQuery 34 │ 1508.04 / 1520.42 ±10.69 / 1539.25 ms │ 1486.04 / 1544.37 ±48.86 / 1630.63 ms │ no change │
│ QQuery 35 │ 289.32 / 332.04 ±29.22 / 369.42 ms │ 292.36 / 324.77 ±34.69 / 389.93 ms │ no change │
│ QQuery 36 │ 65.51 / 70.41 ±5.24 / 79.62 ms │ 63.54 / 66.29 ±2.15 / 69.07 ms │ +1.06x faster │
│ QQuery 37 │ 35.76 / 38.80 ±3.16 / 43.78 ms │ 35.32 / 42.46 ±7.29 / 54.98 ms │ 1.09x slower │
│ QQuery 38 │ 40.98 / 45.55 ±6.46 / 58.36 ms │ 41.42 / 47.29 ±5.62 / 54.15 ms │ no change │
│ QQuery 39 │ 135.23 / 142.74 ±5.79 / 152.85 ms │ 120.20 / 132.07 ±10.80 / 150.86 ms │ +1.08x faster │
│ QQuery 40 │ 14.62 / 14.97 ±0.39 / 15.70 ms │ 14.63 / 17.51 ±5.09 / 27.67 ms │ 1.17x slower │
│ QQuery 41 │ 13.84 / 14.05 ±0.16 / 14.33 ms │ 15.17 / 16.77 ±2.71 / 22.17 ms │ 1.19x slower │
│ QQuery 42 │ 13.46 / 15.11 ±3.01 / 21.12 ms │ 14.66 / 17.05 ±2.57 / 21.83 ms │ 1.13x slower │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20823.44ms │
│ Total Time (row-group-morselization) │ 20609.33ms │
│ Average Time (HEAD) │ 484.27ms │
│ Average Time (row-group-morselization) │ 479.29ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 12 │
│ Queries with No Change │ 27 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 29.8 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1105.3s |
| CPU sys | 65.0s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 31.2 GiB |
| Avg memory | 23.4 GiB |
| CPU user | 1114.3s |
| CPU sys | 67.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291504290
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291385422)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 0.72 / 1.04 ±0.60 / 2.25 ms │ 0.74 / 1.15 ±0.77 / 2.70 ms │ 1.11x slower │
│ QQuery 1 │ 14.22 / 14.58 ±0.50 / 15.56 ms │ 15.29 / 15.60 ±0.31 / 16.18 ms │ 1.07x slower │
│ QQuery 2 │ 42.29 / 42.54 ±0.29 / 43.10 ms │ 36.50 / 37.55 ±0.56 / 38.16 ms │ +1.13x faster │
│ QQuery 3 │ 38.47 / 38.91 ±0.85 / 40.61 ms │ 33.48 / 33.99 ±0.45 / 34.84 ms │ +1.14x faster │
│ QQuery 4 │ 258.55 / 261.12 ±1.89 / 263.78 ms │ 263.78 / 265.97 ±1.70 / 268.61 ms │ no change │
│ QQuery 5 │ 429.87 / 433.18 ±3.05 / 437.93 ms │ 312.34 / 315.07 ±1.59 / 317.24 ms │ +1.37x faster │
│ QQuery 6 │ 6.25 / 6.55 ±0.18 / 6.79 ms │ 8.80 / 13.78 ±3.78 / 20.03 ms │ 2.10x slower │
│ QQuery 7 │ 16.26 / 17.03 ±0.87 / 18.74 ms │ 16.70 / 16.91 ±0.14 / 17.12 ms │ no change │
│ QQuery 8 │ 355.42 / 361.64 ±5.89 / 371.55 ms │ 338.52 / 353.89 ±8.59 / 364.87 ms │ no change │
│ QQuery 9 │ 605.96 / 617.21 ±6.89 / 625.52 ms │ 493.15 / 509.63 ±9.59 / 518.70 ms │ +1.21x faster │
│ QQuery 10 │ 91.43 / 92.19 ±0.68 / 93.06 ms │ 75.45 / 76.30 ±0.61 / 77.16 ms │ +1.21x faster │
│ QQuery 11 │ 105.36 / 107.65 ±3.71 / 115.05 ms │ 86.07 / 87.65 ±1.35 / 90.05 ms │ +1.23x faster │
│ QQuery 12 │ 446.14 / 449.85 ±3.32 / 455.58 ms │ 288.11 / 291.58 ±3.77 / 298.53 ms │ +1.54x faster │
│ QQuery 13 │ 490.60 / 508.13 ±10.20 / 520.24 ms │ 419.34 / 427.49 ±9.11 / 443.68 ms │ +1.19x faster │
│ QQuery 14 │ 437.31 / 445.00 ±6.33 / 456.39 ms │ 299.64 / 303.32 ±2.69 / 307.31 ms │ +1.47x faster │
│ QQuery 15 │ 305.68 / 309.83 ±4.46 / 318.51 ms │ 287.35 / 295.65 ±5.45 / 303.21 ms │ no change │
│ QQuery 16 │ 648.43 / 654.29 ±3.53 / 658.37 ms │ 639.03 / 643.21 ±3.23 / 648.09 ms │ no change │
│ QQuery 17 │ 646.74 / 653.11 ±4.53 / 660.67 ms │ 639.61 / 644.19 ±4.22 / 651.92 ms │ no change │
│ QQuery 18 │ 1284.27 / 1299.74 ±11.53 / 1314.03 ms │ 1278.06 / 1294.25 ±12.08 / 1307.82 ms │ no change │
│ QQuery 19 │ 35.53 / 42.14 ±9.29 / 59.79 ms │ 31.22 / 31.65 ±0.28 / 32.03 ms │ +1.33x faster │
│ QQuery 20 │ 634.72 / 645.89 ±12.03 / 669.14 ms │ 538.91 / 545.49 ±5.92 / 554.99 ms │ +1.18x faster │
│ QQuery 21 │ 713.13 / 726.17 ±9.52 / 738.34 ms │ 638.52 / 650.58 ±13.54 / 676.11 ms │ +1.12x faster │
│ QQuery 22 │ 1391.79 / 1400.01 ±10.88 / 1421.07 ms │ 1202.21 / 1206.18 ±4.41 / 1213.53 ms │ +1.16x faster │
│ QQuery 23 │ 3802.34 / 3827.11 ±16.00 / 3841.63 ms │ 3451.24 / 3481.51 ±17.44 / 3501.77 ms │ +1.10x faster │
│ QQuery 24 │ 215.19 / 218.86 ±3.18 / 224.51 ms │ 177.10 / 180.50 ±3.11 / 184.76 ms │ +1.21x faster │
│ QQuery 25 │ 186.09 / 188.92 ±2.65 / 192.41 ms │ 121.89 / 124.09 ±1.65 / 126.22 ms │ +1.52x faster │
│ QQuery 26 │ 217.01 / 219.83 ±1.81 / 222.54 ms │ 175.92 / 180.35 ±3.89 / 186.27 ms │ +1.22x faster │
│ QQuery 27 │ 757.16 / 768.20 ±9.55 / 780.20 ms │ 682.29 / 699.32 ±13.62 / 716.41 ms │ +1.10x faster │
│ QQuery 28 │ 3455.78 / 3476.75 ±13.88 / 3493.89 ms │ 2931.21 / 2954.37 ±16.78 / 2976.72 ms │ +1.18x faster │
│ QQuery 29 │ 47.07 / 47.63 ±0.59 / 48.39 ms │ 42.20 / 47.75 ±6.34 / 55.75 ms │ no change │
│ QQuery 30 │ 414.16 / 428.73 ±8.87 / 439.99 ms │ 315.49 / 320.93 ±4.64 / 327.78 ms │ +1.34x faster │
│ QQuery 31 │ 384.10 / 397.98 ±10.67 / 416.90 ms │ 305.82 / 320.10 ±10.16 / 337.50 ms │ +1.24x faster │
│ QQuery 32 │ 1029.58 / 1043.19 ±17.24 / 1076.63 ms │ 1015.65 / 1031.84 ±12.43 / 1045.33 ms │ no change │
│ QQuery 33 │ 1488.76 / 1500.61 ±9.67 / 1513.12 ms │ 1495.65 / 1510.62 ±18.59 / 1546.69 ms │ no change │
│ QQuery 34 │ 1480.09 / 1500.25 ±10.69 / 1511.38 ms │ 1509.42 / 1524.96 ±13.53 / 1549.00 ms │ no change │
│ QQuery 35 │ 337.36 / 358.40 ±34.70 / 427.52 ms │ 285.52 / 299.47 ±17.07 / 331.18 ms │ +1.20x faster │
│ QQuery 36 │ 127.25 / 140.16 ±10.63 / 155.10 ms │ 74.79 / 86.51 ±15.39 / 116.47 ms │ +1.62x faster │
│ QQuery 37 │ 54.92 / 55.58 ±0.62 / 56.47 ms │ 46.15 / 48.01 ±1.35 / 49.64 ms │ +1.16x faster │
│ QQuery 38 │ 85.19 / 87.64 ±1.42 / 89.38 ms │ 48.29 / 51.63 ±3.19 / 57.19 ms │ +1.70x faster │
│ QQuery 39 │ 222.85 / 240.90 ±9.27 / 248.11 ms │ 157.31 / 163.87 ±6.78 / 174.93 ms │ +1.47x faster │
│ QQuery 40 │ 21.99 / 27.58 ±5.26 / 36.65 ms │ 17.08 / 17.32 ±0.20 / 17.55 ms │ +1.59x faster │
│ QQuery 41 │ 19.69 / 20.14 ±0.39 / 20.78 ms │ 16.56 / 17.79 ±2.13 / 22.06 ms │ +1.13x faster │
│ QQuery 42 │ 19.59 / 22.61 ±5.04 / 32.67 ms │ 16.26 / 17.13 ±1.57 / 20.25 ms │ +1.32x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 23698.83ms │
│ Total Time (row-group-morselization) │ 21139.15ms │
│ Average Time (HEAD) │ 551.14ms │
│ Average Time (row-group-morselization) │ 491.61ms │
│ Queries Faster │ 29 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 11 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_1 — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 120.0s |
| Peak memory | 30.1 GiB |
| Avg memory | 22.4 GiB |
| CPU user | 1131.2s |
| CPU sys | 67.3s |
| Peak spill | 0 B |
**clickbench_1 — branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 30.6 GiB |
| Avg memory | 22.8 GiB |
| CPU user | 1140.8s |
| CPU sys | 67.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291610041
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291595788-1715-lmg7g 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (88d6ed8bceb8ff74a8701d8290ce238e72ac666c) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..88d6ed8bceb8ff74a8701d8290ce238e72ac666c) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4327840196
> > This looks really cool @Dandandan -- please let me know when it is ready for review (or if you would like help, etc)
>
> I think the perf results look good - I think we mostly need to find out the right API that will also extend to further splitting/merging and other optimizations...
I have an idea in
- https://github.com/apache/datafusion/pull/21852
I probably won't have a chance to work on it until later this week but I think it is showing promise (as it separates IO/CPU in the parquet decoder). @HippoBarro's work upstream in arrow-rs / parquet will likely also make it easier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291256410
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291026396-1703-xbkd8 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291661325
run benchmark clcikbrnch_1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296720389
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4296699660-1748-hqgfb 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (52d6bce6e99b9fba1d98d109fa4b22903340c13d) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..52d6bce6e99b9fba1d98d109fa4b22903340c13d) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4835568792
I also had a PRs here:
- https://github.com/apache/datafusion/pull/21852
Basically what I needed to get excited was some actual performance improvement (just refactoring alone wasn't enough justification for this work)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290802118
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291255796
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291026396-1705-7rg6q 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4814721785
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4819738355
@dandandan just curious did you plan on pushing this forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296721869
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4296699660-1747-s7f9g 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (52d6bce6e99b9fba1d98d109fa4b22903340c13d) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..52d6bce6e99b9fba1d98d109fa4b22903340c13d) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291702516
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.20 / 4.70 ±6.86 / 18.42 ms │ 1.20 / 4.65 ±6.78 / 18.21 ms │ no change │
│ QQuery 1 │ 13.13 / 13.61 ±0.30 / 14.08 ms │ 14.72 / 15.19 ±0.26 / 15.51 ms │ 1.12x slower │
│ QQuery 2 │ 37.40 / 37.79 ±0.33 / 38.39 ms │ 39.45 / 39.86 ±0.35 / 40.36 ms │ 1.05x slower │
│ QQuery 3 │ 31.56 / 31.94 ±0.54 / 32.98 ms │ 34.20 / 34.48 ±0.25 / 34.88 ms │ 1.08x slower │
│ QQuery 4 │ 250.74 / 253.65 ±1.65 / 255.13 ms │ 253.87 / 257.37 ±2.35 / 260.33 ms │ no change │
│ QQuery 5 │ 291.76 / 293.31 ±1.02 / 294.68 ms │ 293.95 / 297.12 ±1.96 / 300.05 ms │ no change │
│ QQuery 6 │ 6.58 / 7.21 ±0.54 / 8.02 ms │ 6.94 / 7.59 ±0.55 / 8.38 ms │ 1.05x slower │
│ QQuery 7 │ 14.52 / 14.69 ±0.12 / 14.86 ms │ 16.66 / 17.12 ±0.30 / 17.57 ms │ 1.16x slower │
│ QQuery 8 │ 342.69 / 346.51 ±2.12 / 348.32 ms │ 340.75 / 343.97 ±3.49 / 350.07 ms │ no change │
│ QQuery 9 │ 511.03 / 530.77 ±10.60 / 538.50 ms │ 520.62 / 535.05 ±9.84 / 550.36 ms │ no change │
│ QQuery 10 │ 76.03 / 76.61 ±0.52 / 77.28 ms │ 77.22 / 78.25 ±0.67 / 78.91 ms │ no change │
│ QQuery 11 │ 87.19 / 88.18 ±0.53 / 88.76 ms │ 89.91 / 91.89 ±1.67 / 94.86 ms │ no change │
│ QQuery 12 │ 285.98 / 289.61 ±3.22 / 294.97 ms │ 285.48 / 290.19 ±4.54 / 296.21 ms │ no change │
│ QQuery 13 │ 410.81 / 419.46 ±8.28 / 434.88 ms │ 411.03 / 418.59 ±8.13 / 432.65 ms │ no change │
│ QQuery 14 │ 296.80 / 299.76 ±2.67 / 304.62 ms │ 300.10 / 305.90 ±6.26 / 314.74 ms │ no change │
│ QQuery 15 │ 297.03 / 301.25 ±5.62 / 312.01 ms │ 296.64 / 301.67 ±4.51 / 309.88 ms │ no change │
│ QQuery 16 │ 643.31 / 649.64 ±6.59 / 661.13 ms │ 637.87 / 647.68 ±5.91 / 655.05 ms │ no change │
│ QQuery 17 │ 645.78 / 649.55 ±1.97 / 651.33 ms │ 643.85 / 650.23 ±3.90 / 655.44 ms │ no change │
│ QQuery 18 │ 1305.06 / 1323.93 ±11.34 / 1335.04 ms │ 1302.06 / 1318.26 ±10.65 / 1332.78 ms │ no change │
│ QQuery 19 │ 29.79 / 30.56 ±0.76 / 31.82 ms │ 30.76 / 31.48 ±0.66 / 32.54 ms │ no change │
│ QQuery 20 │ 520.12 / 526.81 ±6.45 / 538.11 ms │ 497.78 / 500.87 ±2.17 / 504.05 ms │ no change │
│ QQuery 21 │ 595.08 / 602.10 ±6.13 / 609.34 ms │ 583.92 / 585.58 ±1.12 / 586.86 ms │ no change │
│ QQuery 22 │ 1067.98 / 1078.80 ±8.59 / 1090.20 ms │ 1035.40 / 1039.59 ±3.24 / 1045.33 ms │ no change │
│ QQuery 23 │ 3335.57 / 3373.82 ±32.48 / 3420.83 ms │ 3265.86 / 3289.91 ±19.82 / 3322.99 ms │ no change │
│ QQuery 24 │ 42.45 / 44.66 ±3.26 / 51.11 ms │ 46.48 / 52.44 ±8.74 / 69.31 ms │ 1.17x slower │
│ QQuery 25 │ 116.05 / 123.23 ±11.32 / 145.75 ms │ 115.35 / 124.36 ±10.20 / 142.03 ms │ no change │
│ QQuery 26 │ 42.86 / 43.96 ±1.36 / 46.60 ms │ 47.26 / 47.99 ±0.79 / 49.40 ms │ 1.09x slower │
│ QQuery 27 │ 673.21 / 676.98 ±3.90 / 683.86 ms │ 655.30 / 662.87 ±4.12 / 666.79 ms │ no change │
│ QQuery 28 │ 3047.75 / 3056.95 ±8.12 / 3070.11 ms │ 2906.71 / 2911.11 ±5.28 / 2921.02 ms │ no change │
│ QQuery 29 │ 42.88 / 43.11 ±0.14 / 43.23 ms │ 45.60 / 56.35 ±18.98 / 94.28 ms │ 1.31x slower │
│ QQuery 30 │ 319.98 / 323.46 ±2.90 / 328.22 ms │ 316.83 / 324.51 ±6.18 / 334.12 ms │ no change │
│ QQuery 31 │ 309.27 / 314.53 ±4.02 / 320.16 ms │ 309.16 / 323.23 ±11.78 / 343.55 ms │ no change │
│ QQuery 32 │ 1048.53 / 1055.19 ±6.81 / 1067.93 ms │ 1035.75 / 1045.69 ±7.70 / 1054.22 ms │ no change │
│ QQuery 33 │ 1468.70 / 1492.42 ±16.75 / 1514.33 ms │ 1515.30 / 1533.32 ±24.25 / 1581.33 ms │ no change │
│ QQuery 34 │ 1486.32 / 1522.04 ±29.84 / 1567.98 ms │ 1509.94 / 1521.73 ±9.98 / 1533.90 ms │ no change │
│ QQuery 35 │ 306.48 / 329.24 ±37.72 / 404.42 ms │ 306.88 / 328.96 ±29.92 / 387.49 ms │ no change │
│ QQuery 36 │ 67.82 / 76.00 ±9.19 / 91.19 ms │ 64.97 / 69.13 ±3.65 / 74.85 ms │ +1.10x faster │
│ QQuery 37 │ 36.63 / 39.52 ±2.83 / 43.19 ms │ 37.15 / 37.38 ±0.25 / 37.85 ms │ +1.06x faster │
│ QQuery 38 │ 44.68 / 47.30 ±2.47 / 51.22 ms │ 39.77 / 48.23 ±6.45 / 59.16 ms │ no change │
│ QQuery 39 │ 125.81 / 137.58 ±6.63 / 145.90 ms │ 138.51 / 142.56 ±3.06 / 146.95 ms │ no change │
│ QQuery 40 │ 15.43 / 17.64 ±2.56 / 21.30 ms │ 15.12 / 15.30 ±0.15 / 15.56 ms │ +1.15x faster │
│ QQuery 41 │ 14.57 / 15.60 ±1.36 / 18.18 ms │ 14.47 / 15.27 ±1.37 / 18.01 ms │ no change │
│ QQuery 42 │ 14.19 / 15.41 ±1.75 / 18.88 ms │ 14.11 / 16.10 ±3.64 / 23.38 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20619.08ms │
│ Total Time (row-group-morselization) │ 20378.99ms │
│ Average Time (HEAD) │ 479.51ms │
│ Average Time (row-group-morselization) │ 473.93ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 8 │
│ Queries with No Change │ 32 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 29.8 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1097.9s |
| CPU sys | 62.2s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 29.4 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1104.7s |
| CPU sys | 63.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291938395
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4293208356
FYI tpch will continue to fail until https://github.com/apache/datafusion/issues/21625 is resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297908241
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4297752534)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 40.77 / 41.86 ±1.09 / 43.41 ms │ 40.55 / 41.53 ±0.90 / 42.83 ms │ no change │
│ QQuery 2 │ 21.03 / 21.61 ±0.61 / 22.60 ms │ 21.05 / 21.35 ±0.21 / 21.61 ms │ no change │
│ QQuery 3 │ 38.27 / 40.28 ±2.15 / 43.87 ms │ 38.41 / 39.22 ±0.48 / 39.72 ms │ no change │
│ QQuery 4 │ 18.56 / 18.78 ±0.38 / 19.54 ms │ 18.65 / 19.14 ±0.25 / 19.40 ms │ no change │
│ QQuery 5 │ 48.15 / 50.33 ±1.65 / 52.43 ms │ 48.44 / 49.37 ±1.18 / 51.71 ms │ no change │
│ QQuery 6 │ 17.31 / 17.52 ±0.20 / 17.86 ms │ 18.02 / 18.22 ±0.17 / 18.43 ms │ no change │
│ QQuery 7 │ 54.22 / 55.36 ±1.11 / 57.28 ms │ 54.52 / 55.60 ±0.76 / 56.68 ms │ no change │
│ QQuery 8 │ 47.91 / 48.23 ±0.21 / 48.54 ms │ 48.70 / 48.95 ±0.14 / 49.12 ms │ no change │
│ QQuery 9 │ 53.58 / 55.19 ±1.88 / 58.85 ms │ 53.48 / 53.78 ±0.25 / 54.10 ms │ no change │
│ QQuery 10 │ 66.07 / 66.29 ±0.18 / 66.50 ms │ 66.88 / 67.40 ±0.27 / 67.66 ms │ no change │
│ QQuery 11 │ 14.36 / 14.90 ±0.84 / 16.57 ms │ 14.18 / 14.36 ±0.16 / 14.65 ms │ no change │
│ QQuery 12 │ 27.86 / 28.27 ±0.53 / 29.25 ms │ 27.82 / 28.19 ±0.45 / 29.07 ms │ no change │
│ QQuery 13 │ 37.29 / 38.28 ±0.73 / 39.31 ms │ 37.03 / 37.29 ±0.22 / 37.60 ms │ no change │
│ QQuery 14 │ 28.08 / 28.44 ±0.35 / 29.10 ms │ 28.85 / 29.03 ±0.15 / 29.25 ms │ no change │
│ QQuery 15 │ 33.51 / 34.05 ±0.75 / 35.52 ms │ 34.90 / 35.04 ±0.12 / 35.23 ms │ no change │
│ QQuery 16 │ 15.30 / 15.91 ±0.78 / 17.46 ms │ 15.46 / 15.50 ±0.03 / 15.53 ms │ no change │
│ QQuery 17 │ 79.85 / 81.18 ±1.65 / 84.32 ms │ 79.13 / 79.84 ±0.39 / 80.30 ms │ no change │
│ QQuery 18 │ 76.34 / 76.85 ±0.54 / 77.79 ms │ 74.84 / 75.67 ±0.44 / 76.14 ms │ no change │
│ QQuery 19 │ 37.82 / 38.12 ±0.29 / 38.63 ms │ 37.98 / 38.38 ±0.40 / 39.05 ms │ no change │
│ QQuery 20 │ 39.57 / 40.60 ±1.23 / 43.00 ms │ 40.38 / 40.86 ±0.40 / 41.59 ms │ no change │
│ QQuery 21 │ 62.19 / 65.21 ±1.75 / 66.78 ms │ 63.04 / 63.60 ±0.62 / 64.75 ms │ no change │
│ QQuery 22 │ 17.48 / 17.63 ±0.19 / 18.00 ms │ 17.30 / 17.43 ±0.10 / 17.54 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 894.89ms │
│ Total Time (row-group-morselization) │ 889.74ms │
│ Average Time (HEAD) │ 40.68ms │
│ Average Time (row-group-morselization) │ 40.44ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.4 GiB |
| Avg memory | 4.8 GiB |
| CPU user | 33.8s |
| CPU sys | 2.5s |
| Peak spill | 0 B |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.5 GiB |
| Avg memory | 4.8 GiB |
| CPU user | 34.0s |
| CPU sys | 2.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297668551
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297782717
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4297752534)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4297752534-1755-2lpf9 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (a5c02d8a40de9f8cb0e16d80e596c2c03715e961) to 4bff17e (merge-base) [diff](https://github.com/apache/datafusion/compare/4bff17ebd2394e319b4adee77df18a011116ff2d..a5c02d8a40de9f8cb0e16d80e596c2c03715e961) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291255769
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291134705-1701-2gh92 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291527520
run benchmark clickbench_1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292141029
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292835804
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.18 / 4.65 ±6.79 / 18.22 ms │ 1.16 / 4.66 ±6.85 / 18.35 ms │ no change │
│ QQuery 1 │ 12.77 / 13.25 ±0.35 / 13.83 ms │ 14.51 / 14.87 ±0.19 / 15.09 ms │ 1.12x slower │
│ QQuery 2 │ 36.83 / 37.51 ±0.51 / 38.14 ms │ 39.17 / 39.70 ±0.43 / 40.21 ms │ 1.06x slower │
│ QQuery 3 │ 31.23 / 31.82 ±0.73 / 33.23 ms │ 33.44 / 34.17 ±0.55 / 35.05 ms │ 1.07x slower │
│ QQuery 4 │ 242.36 / 247.90 ±3.24 / 251.44 ms │ 248.88 / 254.72 ±4.30 / 261.58 ms │ no change │
│ QQuery 5 │ 287.55 / 290.17 ±2.67 / 294.81 ms │ 290.25 / 293.71 ±2.01 / 295.52 ms │ no change │
│ QQuery 6 │ 7.02 / 7.92 ±1.32 / 10.50 ms │ 6.63 / 7.44 ±0.41 / 7.77 ms │ +1.06x faster │
│ QQuery 7 │ 14.63 / 14.73 ±0.06 / 14.80 ms │ 16.08 / 16.35 ±0.18 / 16.59 ms │ 1.11x slower │
│ QQuery 8 │ 335.15 / 338.14 ±2.27 / 340.86 ms │ 338.95 / 343.87 ±5.70 / 354.98 ms │ no change │
│ QQuery 9 │ 497.32 / 513.21 ±13.79 / 530.54 ms │ 497.60 / 517.81 ±13.88 / 535.07 ms │ no change │
│ QQuery 10 │ 74.92 / 76.55 ±0.94 / 77.70 ms │ 76.99 / 77.86 ±0.87 / 78.93 ms │ no change │
│ QQuery 11 │ 86.70 / 88.85 ±3.32 / 95.47 ms │ 87.56 / 88.93 ±0.84 / 89.93 ms │ no change │
│ QQuery 12 │ 278.13 / 284.04 ±4.21 / 289.11 ms │ 279.90 / 285.32 ±3.37 / 289.80 ms │ no change │
│ QQuery 13 │ 403.87 / 411.90 ±6.56 / 418.31 ms │ 401.95 / 410.48 ±6.28 / 421.24 ms │ no change │
│ QQuery 14 │ 292.43 / 295.85 ±2.61 / 299.00 ms │ 291.49 / 293.51 ±1.86 / 296.37 ms │ no change │
│ QQuery 15 │ 288.81 / 294.60 ±3.72 / 298.97 ms │ 291.95 / 299.84 ±6.90 / 308.70 ms │ no change │
│ QQuery 16 │ 634.30 / 639.32 ±3.69 / 644.11 ms │ 635.42 / 641.41 ±8.56 / 658.32 ms │ no change │
│ QQuery 17 │ 636.57 / 641.98 ±4.39 / 646.32 ms │ 638.79 / 645.01 ±5.40 / 653.85 ms │ no change │
│ QQuery 18 │ 1276.43 / 1302.08 ±19.30 / 1330.48 ms │ 1286.49 / 1307.57 ±15.82 / 1331.98 ms │ no change │
│ QQuery 19 │ 28.93 / 31.86 ±3.76 / 38.66 ms │ 30.79 / 31.77 ±0.53 / 32.24 ms │ no change │
│ QQuery 20 │ 515.99 / 522.23 ±6.63 / 534.29 ms │ 493.76 / 506.95 ±19.03 / 544.69 ms │ no change │
│ QQuery 21 │ 599.25 / 604.86 ±2.95 / 607.94 ms │ 579.12 / 584.15 ±3.33 / 588.65 ms │ no change │
│ QQuery 22 │ 1065.65 / 1075.95 ±6.19 / 1084.98 ms │ 1029.58 / 1041.44 ±9.85 / 1059.11 ms │ no change │
│ QQuery 23 │ 3338.39 / 3367.36 ±18.06 / 3389.94 ms │ 3236.06 / 3256.54 ±18.75 / 3285.22 ms │ no change │
│ QQuery 24 │ 42.18 / 42.63 ±0.63 / 43.83 ms │ 46.39 / 48.36 ±3.11 / 54.54 ms │ 1.13x slower │
│ QQuery 25 │ 116.31 / 118.14 ±1.73 / 121.11 ms │ 115.17 / 118.97 ±5.13 / 128.88 ms │ no change │
│ QQuery 26 │ 42.54 / 43.95 ±1.65 / 46.91 ms │ 46.84 / 47.09 ±0.19 / 47.31 ms │ 1.07x slower │
│ QQuery 27 │ 672.93 / 676.35 ±3.10 / 681.15 ms │ 651.99 / 662.67 ±6.66 / 670.22 ms │ no change │
│ QQuery 28 │ 3021.91 / 3041.47 ±10.57 / 3050.69 ms │ 2876.60 / 2884.81 ±4.88 / 2891.15 ms │ +1.05x faster │
│ QQuery 29 │ 43.05 / 50.47 ±14.38 / 79.22 ms │ 45.79 / 48.16 ±2.86 / 53.38 ms │ no change │
│ QQuery 30 │ 315.48 / 318.95 ±2.25 / 321.52 ms │ 314.30 / 320.49 ±3.60 / 324.73 ms │ no change │
│ QQuery 31 │ 303.05 / 311.29 ±4.62 / 314.99 ms │ 309.71 / 313.60 ±3.15 / 317.70 ms │ no change │
│ QQuery 32 │ 1020.34 / 1043.73 ±20.99 / 1083.13 ms │ 1028.85 / 1039.59 ±12.02 / 1061.60 ms │ no change │
│ QQuery 33 │ 1458.59 / 1484.82 ±14.21 / 1496.64 ms │ 1472.76 / 1490.99 ±10.11 / 1502.19 ms │ no change │
│ QQuery 34 │ 1483.77 / 1503.37 ±21.64 / 1535.16 ms │ 1472.77 / 1504.44 ±23.26 / 1541.91 ms │ no change │
│ QQuery 35 │ 295.52 / 307.49 ±12.98 / 331.17 ms │ 299.40 / 324.37 ±22.75 / 353.06 ms │ 1.05x slower │
│ QQuery 36 │ 63.37 / 67.12 ±3.09 / 71.41 ms │ 64.07 / 70.20 ±6.47 / 80.15 ms │ no change │
│ QQuery 37 │ 36.18 / 38.28 ±3.69 / 45.67 ms │ 39.09 / 41.73 ±1.84 / 44.12 ms │ 1.09x slower │
│ QQuery 38 │ 41.54 / 46.17 ±3.75 / 51.87 ms │ 39.23 / 42.74 ±4.80 / 52.04 ms │ +1.08x faster │
│ QQuery 39 │ 124.15 / 132.74 ±6.58 / 143.82 ms │ 132.69 / 139.85 ±5.99 / 147.96 ms │ 1.05x slower │
│ QQuery 40 │ 14.75 / 15.07 ±0.21 / 15.39 ms │ 15.32 / 18.08 ±3.17 / 24.25 ms │ 1.20x slower │
│ QQuery 41 │ 14.28 / 16.70 ±3.14 / 22.73 ms │ 14.63 / 14.86 ±0.17 / 15.03 ms │ +1.12x faster │
│ QQuery 42 │ 13.81 / 16.90 ±3.28 / 21.50 ms │ 13.97 / 14.17 ±0.18 / 14.47 ms │ +1.19x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 20412.37ms │
│ Total Time (row-group-morselization) │ 20143.24ms │
│ Average Time (HEAD) │ 474.71ms │
│ Average Time (row-group-morselization) │ 468.45ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 10 │
│ Queries with No Change │ 28 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.4 GiB |
| Avg memory | 23.2 GiB |
| CPU user | 1085.0s |
| CPU sys | 62.8s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.4 GiB |
| Avg memory | 23.2 GiB |
| CPU user | 1088.7s |
| CPU sys | 64.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291610688
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291595788-1716-kkzrg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (88d6ed8bceb8ff74a8701d8290ce238e72ac666c) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..88d6ed8bceb8ff74a8701d8290ce238e72ac666c) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291708572
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4291661325) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
struct_query_sql
substr
substr_index
substring
sum
to_char
to_hex
to_local_time
to_time
to_timestamp
topk_aggregate
topk_repartition
translate
trim
trunc
unhex
upper
uuid
window_query_sql
with_hashes
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4293170422
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291352119
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.72 / 7.25 ±0.87 / 8.98 ms │ 7.26 / 7.71 ±0.73 / 9.16 ms │ 1.06x slower │
│ QQuery 2 │ 147.00 / 148.96 ±2.43 / 153.54 ms │ 144.35 / 145.43 ±0.96 / 147.22 ms │ no change │
│ QQuery 3 │ 112.44 / 113.35 ±0.83 / 114.76 ms │ 112.57 / 114.10 ±1.01 / 115.30 ms │ no change │
│ QQuery 4 │ 1301.57 / 1344.81 ±29.14 / 1392.29 ms │ 1370.20 / 1428.81 ±32.41 / 1470.13 ms │ 1.06x slower │
│ QQuery 5 │ 172.26 / 174.32 ±1.68 / 176.15 ms │ 175.06 / 177.63 ±2.26 / 181.16 ms │ no change │
│ QQuery 6 │ 840.07 / 859.53 ±29.19 / 917.67 ms │ 861.35 / 882.13 ±18.62 / 908.28 ms │ no change │
│ QQuery 7 │ 332.54 / 336.78 ±3.90 / 343.49 ms │ 334.02 / 341.50 ±4.20 / 345.41 ms │ no change │
│ QQuery 8 │ 112.46 / 114.92 ±1.74 / 116.75 ms │ 112.75 / 114.20 ±1.63 / 117.37 ms │ no change │
│ QQuery 9 │ 101.43 / 109.90 ±9.16 / 127.31 ms │ 100.65 / 103.99 ±3.82 / 110.08 ms │ +1.06x faster │
│ QQuery 10 │ 101.20 / 104.05 ±1.79 / 106.55 ms │ 101.93 / 104.87 ±3.89 / 112.44 ms │ no change │
│ QQuery 11 │ 890.52 / 940.71 ±47.49 / 1029.79 ms │ 912.43 / 923.02 ±6.34 / 931.86 ms │ no change │
│ QQuery 12 │ 42.90 / 43.90 ±0.86 / 44.98 ms │ 43.46 / 43.92 ±0.33 / 44.35 ms │ no change │
│ QQuery 13 │ 386.48 / 394.44 ±6.82 / 405.92 ms │ 386.36 / 395.62 ±7.13 / 405.85 ms │ no change │
│ QQuery 14 │ 982.37 / 986.61 ±2.89 / 990.83 ms │ 978.91 / 993.72 ±12.19 / 1014.43 ms │ no change │
│ QQuery 15 │ 14.67 / 14.79 ±0.18 / 15.13 ms │ 15.70 / 16.04 ±0.19 / 16.24 ms │ 1.09x slower │
│ QQuery 16 │ 7.28 / 7.48 ±0.15 / 7.74 ms │ 7.99 / 8.09 ±0.09 / 8.23 ms │ 1.08x slower │
│ QQuery 17 │ 220.97 / 224.67 ±3.96 / 231.47 ms │ 224.62 / 230.62 ±4.88 / 236.35 ms │ no change │
│ QQuery 18 │ 123.42 / 125.95 ±1.99 / 128.16 ms │ 122.15 / 125.07 ±2.53 / 129.00 ms │ no change │
│ QQuery 19 │ 152.97 / 157.36 ±4.93 / 163.90 ms │ 152.87 / 155.96 ±2.31 / 159.52 ms │ no change │
│ QQuery 20 │ 12.79 / 13.47 ±0.68 / 14.60 ms │ 14.11 / 14.25 ±0.08 / 14.33 ms │ 1.06x slower │
│ QQuery 21 │ 18.75 / 18.89 ±0.11 / 19.05 ms │ 20.26 / 20.63 ±0.22 / 20.88 ms │ 1.09x slower │
│ QQuery 22 │ 472.40 / 483.39 ±12.48 / 500.26 ms │ 484.16 / 490.97 ±6.87 / 502.51 ms │ no change │
│ QQuery 23 │ 819.99 / 856.98 ±21.64 / 884.94 ms │ 832.64 / 844.22 ±9.86 / 856.58 ms │ no change │
│ QQuery 24 │ 374.84 / 388.09 ±9.57 / 400.63 ms │ 379.71 / 387.02 ±6.25 / 395.91 ms │ no change │
│ QQuery 25 │ 332.89 / 337.90 ±4.33 / 344.75 ms │ 333.43 / 342.76 ±5.69 / 350.54 ms │ no change │
│ QQuery 26 │ 78.65 / 79.10 ±0.26 / 79.45 ms │ 77.66 / 79.24 ±1.42 / 81.34 ms │ no change │
│ QQuery 27 │ 7.21 / 7.40 ±0.11 / 7.51 ms │ 7.37 / 7.46 ±0.08 / 7.60 ms │ no change │
│ QQuery 28 │ 152.99 / 154.60 ±1.83 / 158.13 ms │ 148.58 / 151.17 ±2.38 / 154.30 ms │ no change │
│ QQuery 29 │ 281.62 / 287.78 ±5.06 / 296.76 ms │ 273.57 / 283.22 ±6.55 / 290.37 ms │ no change │
│ QQuery 30 │ 43.21 / 43.88 ±0.79 / 45.39 ms │ 41.64 / 42.12 ±0.48 / 42.95 ms │ no change │
│ QQuery 31 │ 168.32 / 170.34 ±1.34 / 172.04 ms │ 164.25 / 167.12 ±1.91 / 170.20 ms │ no change │
│ QQuery 32 │ 14.33 / 14.53 ±0.17 / 14.78 ms │ 13.33 / 13.57 ±0.25 / 14.05 ms │ +1.07x faster │
│ QQuery 33 │ 142.37 / 143.12 ±0.63 / 144.15 ms │ 137.87 / 140.96 ±2.25 / 144.08 ms │ no change │
│ QQuery 34 │ 7.42 / 7.52 ±0.07 / 7.60 ms │ 7.47 / 7.61 ±0.09 / 7.74 ms │ no change │
│ QQuery 35 │ 102.59 / 104.51 ±1.63 / 107.22 ms │ 99.84 / 101.76 ±2.31 / 106.03 ms │ no change │
│ QQuery 36 │ 7.08 / 7.24 ±0.17 / 7.48 ms │ 6.59 / 6.75 ±0.15 / 6.96 ms │ +1.07x faster │
│ QQuery 37 │ 8.70 / 8.81 ±0.11 / 9.01 ms │ 8.19 / 8.28 ±0.06 / 8.37 ms │ +1.06x faster │
│ QQuery 38 │ 88.82 / 90.80 ±1.62 / 93.75 ms │ 90.15 / 91.67 ±1.69 / 94.96 ms │ no change │
│ QQuery 39 │ 127.23 / 129.42 ±1.19 / 130.60 ms │ 116.36 / 120.01 ±3.50 / 124.51 ms │ +1.08x faster │
│ QQuery 40 │ 106.69 / 113.99 ±5.11 / 119.75 ms │ 102.16 / 112.01 ±5.98 / 118.34 ms │ no change │
│ QQuery 41 │ 14.92 / 15.03 ±0.09 / 15.19 ms │ 14.00 / 14.17 ±0.26 / 14.70 ms │ +1.06x faster │
│ QQuery 42 │ 108.29 / 110.93 ±4.24 / 119.36 ms │ 105.26 / 108.78 ±3.79 / 115.88 ms │ no change │
│ QQuery 43 │ 6.12 / 6.19 ±0.05 / 6.24 ms │ 6.13 / 6.28 ±0.11 / 6.45 ms │ no change │
│ QQuery 44 │ 12.29 / 12.36 ±0.04 / 12.39 ms │ 12.93 / 12.99 ±0.06 / 13.11 ms │ 1.05x slower │
│ QQuery 45 │ 49.43 / 50.09 ±0.51 / 50.98 ms │ 50.81 / 51.24 ±0.31 / 51.68 ms │ no change │
│ QQuery 46 │ 8.27 / 8.44 ±0.20 / 8.83 ms │ 8.57 / 8.75 ±0.19 / 9.11 ms │ no change │
│ QQuery 47 │ 714.16 / 765.04 ±29.35 / 802.87 ms │ 713.81 / 732.01 ±16.87 / 755.31 ms │ no change │
│ QQuery 48 │ 271.74 / 280.82 ±8.32 / 294.11 ms │ 271.61 / 280.31 ±5.67 / 287.94 ms │ no change │
│ QQuery 49 │ 248.31 / 251.25 ±3.28 / 256.59 ms │ 249.90 / 251.43 ±0.87 / 252.31 ms │ no change │
│ QQuery 50 │ 200.12 / 212.65 ±10.42 / 225.60 ms │ 206.63 / 220.53 ±7.48 / 227.78 ms │ no change │
│ QQuery 51 │ 176.20 / 179.10 ±3.09 / 184.51 ms │ 175.74 / 177.86 ±1.75 / 180.04 ms │ no change │
│ QQuery 52 │ 106.24 / 108.56 ±2.49 / 113.23 ms │ 106.61 / 107.86 ±1.72 / 111.15 ms │ no change │
│ QQuery 53 │ 101.60 / 102.18 ±0.50 / 102.85 ms │ 103.02 / 104.50 ±1.23 / 106.35 ms │ no change │
│ QQuery 54 │ 143.65 / 147.54 ±3.08 / 152.16 ms │ 148.34 / 149.92 ±1.22 / 151.79 ms │ no change │
│ QQuery 55 │ 105.65 / 107.27 ±1.33 / 108.77 ms │ 106.92 / 108.41 ±1.14 / 109.95 ms │ no change │
│ QQuery 56 │ 138.81 / 141.10 ±2.97 / 146.83 ms │ 138.86 / 139.87 ±0.99 / 141.26 ms │ no change │
│ QQuery 57 │ 164.71 / 168.90 ±4.61 / 177.16 ms │ 165.46 / 168.47 ±1.70 / 170.44 ms │ no change │
│ QQuery 58 │ 310.26 / 312.20 ±2.04 / 315.77 ms │ 315.13 / 317.56 ±1.96 / 320.57 ms │ no change │
│ QQuery 59 │ 195.51 / 199.68 ±3.96 / 206.16 ms │ 202.05 / 204.04 ±1.71 / 207.03 ms │ no change │
│ QQuery 60 │ 139.33 / 141.09 ±1.04 / 142.21 ms │ 143.89 / 145.44 ±1.01 / 146.85 ms │ no change │
│ QQuery 61 │ 13.74 / 13.96 ±0.14 / 14.12 ms │ 14.15 / 14.25 ±0.12 / 14.43 ms │ no change │
│ QQuery 62 │ 904.73 / 918.22 ±9.80 / 932.05 ms │ 916.60 / 925.80 ±6.20 / 933.02 ms │ no change │
│ QQuery 63 │ 101.24 / 103.65 ±1.51 / 105.59 ms │ 102.83 / 105.25 ±1.28 / 106.56 ms │ no change │
│ QQuery 64 │ 663.33 / 677.82 ±17.04 / 710.23 ms │ 679.36 / 690.90 ±8.21 / 704.02 ms │ no change │
│ QQuery 65 │ 240.18 / 255.20 ±11.58 / 268.75 ms │ 245.66 / 254.84 ±8.87 / 268.56 ms │ no change │
│ QQuery 66 │ 212.34 / 224.35 ±11.12 / 241.98 ms │ 214.52 / 226.01 ±12.30 / 247.84 ms │ no change │
│ QQuery 67 │ 288.49 / 306.64 ±15.23 / 329.33 ms │ 290.39 / 308.84 ±15.92 / 332.90 ms │ no change │
│ QQuery 68 │ 9.20 / 9.35 ±0.13 / 9.53 ms │ 9.03 / 9.26 ±0.21 / 9.53 ms │ no change │
│ QQuery 69 │ 97.01 / 100.50 ±4.38 / 109.05 ms │ 97.41 / 98.15 ±0.57 / 99.11 ms │ no change │
│ QQuery 70 │ 315.05 / 329.60 ±16.41 / 352.89 ms │ 311.43 / 325.08 ±8.66 / 336.22 ms │ no change │
│ QQuery 71 │ 133.50 / 135.07 ±1.04 / 136.70 ms │ 132.02 / 134.09 ±2.00 / 137.26 ms │ no change │
│ QQuery 72 │ 600.24 / 622.88 ±15.14 / 643.48 ms │ 587.49 / 600.22 ±11.27 / 617.95 ms │ no change │
│ QQuery 73 │ 6.61 / 6.80 ±0.24 / 7.27 ms │ 7.27 / 7.37 ±0.07 / 7.48 ms │ 1.08x slower │
│ QQuery 74 │ 556.73 / 593.88 ±21.03 / 611.89 ms │ 575.69 / 621.74 ±37.53 / 657.85 ms │ no change │
│ QQuery 75 │ 269.70 / 272.30 ±2.17 / 275.15 ms │ 267.51 / 272.12 ±4.37 / 277.90 ms │ no change │
│ QQuery 76 │ 131.55 / 134.37 ±2.70 / 138.73 ms │ 130.03 / 135.21 ±3.02 / 139.08 ms │ no change │
│ QQuery 77 │ 187.98 / 190.25 ±1.41 / 191.91 ms │ 186.14 / 190.49 ±3.65 / 196.20 ms │ no change │
│ QQuery 78 │ 330.21 / 334.97 ±4.67 / 342.92 ms │ 329.77 / 338.57 ±9.72 / 352.35 ms │ no change │
│ QQuery 79 │ 228.09 / 239.95 ±11.69 / 254.41 ms │ 225.47 / 235.52 ±11.52 / 257.32 ms │ no change │
│ QQuery 80 │ 318.93 / 321.03 ±1.90 / 323.99 ms │ 318.39 / 322.48 ±3.84 / 327.74 ms │ no change │
│ QQuery 81 │ 26.03 / 26.83 ±0.87 / 28.37 ms │ 27.30 / 27.91 ±1.00 / 29.91 ms │ no change │
│ QQuery 82 │ 40.85 / 41.14 ±0.23 / 41.50 ms │ 39.45 / 39.94 ±0.64 / 41.17 ms │ no change │
│ QQuery 83 │ 37.35 / 38.65 ±0.71 / 39.39 ms │ 36.29 / 36.81 ±0.33 / 37.27 ms │ no change │
│ QQuery 84 │ 45.97 / 46.11 ±0.16 / 46.42 ms │ 45.79 / 46.03 ±0.17 / 46.18 ms │ no change │
│ QQuery 85 │ 140.03 / 141.10 ±0.93 / 142.61 ms │ 139.13 / 142.40 ±2.55 / 145.90 ms │ no change │
│ QQuery 86 │ 37.83 / 38.35 ±0.39 / 38.89 ms │ 37.12 / 37.40 ±0.40 / 38.17 ms │ no change │
│ QQuery 87 │ 3.70 / 3.79 ±0.10 / 3.96 ms │ 3.49 / 3.58 ±0.12 / 3.82 ms │ +1.06x faster │
│ QQuery 88 │ 101.34 / 102.02 ±0.82 / 103.62 ms │ 99.34 / 102.50 ±2.67 / 106.09 ms │ no change │
│ QQuery 89 │ 117.57 / 121.15 ±5.88 / 132.86 ms │ 115.99 / 117.03 ±0.73 / 117.87 ms │ no change │
│ QQuery 90 │ 22.84 / 23.17 ±0.27 / 23.57 ms │ 21.78 / 22.29 ±0.37 / 22.71 ms │ no change │
│ QQuery 91 │ 58.71 / 59.36 ±0.50 / 59.94 ms │ 59.09 / 61.22 ±1.36 / 63.18 ms │ no change │
│ QQuery 92 │ 55.65 / 56.46 ±0.60 / 57.19 ms │ 55.69 / 56.22 ±0.58 / 57.32 ms │ no change │
│ QQuery 93 │ 179.75 / 182.86 ±3.60 / 189.63 ms │ 181.25 / 185.73 ±4.89 / 192.60 ms │ no change │
│ QQuery 94 │ 59.76 / 60.25 ±0.29 / 60.57 ms │ 59.96 / 60.46 ±0.43 / 61.19 ms │ no change │
│ QQuery 95 │ 127.53 / 127.98 ±0.38 / 128.68 ms │ 126.81 / 128.55 ±1.20 / 130.26 ms │ no change │
│ QQuery 96 │ 68.08 / 69.24 ±0.88 / 70.71 ms │ 67.79 / 68.74 ±1.02 / 70.53 ms │ no change │
│ QQuery 97 │ 116.56 / 120.49 ±3.35 / 125.32 ms │ 117.44 / 124.57 ±3.87 / 128.28 ms │ no change │
│ QQuery 98 │ 149.80 / 152.24 ±2.20 / 155.71 ms │ 157.39 / 159.23 ±1.35 / 161.52 ms │ no change │
│ QQuery 99 │ 10961.66 / 11025.99 ±43.05 / 11091.35 ms │ 10882.92 / 10985.28 ±90.19 / 11091.59 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 31219.91ms │
│ Total Time (row-group-morselization) │ 31277.71ms │
│ Average Time (HEAD) │ 315.35ms │
│ Average Time (row-group-morselization) │ 315.94ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 8 │
│ Queries with No Change │ 84 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 261.9s |
| CPU sys | 8.7s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.3 GiB |
| Avg memory | 5.6 GiB |
| CPU user | 262.0s |
| CPU sys | 8.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294420644
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4294343524)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpch_sf10.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 328.05 / 330.49 ±1.60 / 332.91 ms │ 329.76 / 330.88 ±0.97 / 332.26 ms │ no change │
│ QQuery 2 │ 136.25 / 139.09 ±2.73 / 144.06 ms │ 135.04 / 136.62 ±1.16 / 138.02 ms │ no change │
│ QQuery 3 │ 290.71 / 296.70 ±3.26 / 300.44 ms │ 287.84 / 291.20 ±2.70 / 295.80 ms │ no change │
│ QQuery 4 │ 154.24 / 155.02 ±1.12 / 157.16 ms │ 156.09 / 157.48 ±0.88 / 158.70 ms │ no change │
│ QQuery 5 │ 427.45 / 432.77 ±4.06 / 439.01 ms │ 423.54 / 424.22 ±0.71 / 425.58 ms │ no change │
│ QQuery 6 │ 132.60 / 134.39 ±1.47 / 136.88 ms │ 136.90 / 137.82 ±0.81 / 139.06 ms │ no change │
│ QQuery 7 │ 543.34 / 547.12 ±4.67 / 555.78 ms │ 536.39 / 540.55 ±2.99 / 545.61 ms │ no change │
│ QQuery 8 │ 462.77 / 467.13 ±2.33 / 469.70 ms │ 458.93 / 461.59 ±2.95 / 467.29 ms │ no change │
│ QQuery 9 │ 652.12 / 657.60 ±4.58 / 665.84 ms │ 643.96 / 652.89 ±7.10 / 665.05 ms │ no change │
│ QQuery 10 │ 330.76 / 342.07 ±6.07 / 348.28 ms │ 330.80 / 333.03 ±1.59 / 335.09 ms │ no change │
│ QQuery 11 │ 105.07 / 110.54 ±7.90 / 126.15 ms │ 102.92 / 105.61 ±3.74 / 113.04 ms │ no change │
│ QQuery 12 │ 200.14 / 205.38 ±5.73 / 213.13 ms │ 203.95 / 205.86 ±2.06 / 209.78 ms │ no change │
│ QQuery 13 │ 319.44 / 322.39 ±4.76 / 331.79 ms │ 297.38 / 298.85 ±1.86 / 302.24 ms │ +1.08x faster │
│ QQuery 14 │ 183.59 / 187.12 ±3.27 / 191.15 ms │ 187.09 / 188.06 ±0.94 / 189.81 ms │ no change │
│ QQuery 15 │ 332.26 / 336.53 ±4.04 / 344.21 ms │ 336.08 / 337.90 ±1.58 / 340.75 ms │ no change │
│ QQuery 16 │ 79.16 / 83.63 ±4.12 / 91.34 ms │ 79.71 / 83.38 ±6.90 / 97.17 ms │ no change │
│ QQuery 17 │ 754.73 / 766.18 ±9.77 / 783.96 ms │ 742.90 / 746.00 ±2.28 / 749.25 ms │ no change │
│ QQuery 18 │ 770.05 / 792.19 ±18.30 / 819.67 ms │ 765.83 / 782.54 ±19.75 / 820.12 ms │ no change │
│ QQuery 19 │ 270.85 / 293.73 ±34.50 / 361.56 ms │ 270.80 / 273.10 ±2.58 / 278.13 ms │ +1.08x faster │
│ QQuery 20 │ 307.33 / 318.76 ±7.05 / 326.97 ms │ 298.38 / 303.27 ±4.01 / 309.41 ms │ no change │
│ QQuery 21 │ 820.13 / 824.46 ±3.54 / 830.61 ms │ 799.32 / 804.21 ±3.66 / 810.28 ms │ no change │
│ QQuery 22 │ 82.54 / 84.31 ±1.96 / 87.71 ms │ 79.38 / 80.30 ±0.86 / 81.56 ms │ no change │
└───────────┴────────────────────────────────────┴────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 7827.62ms │
│ Total Time (row-group-morselization) │ 7675.37ms │
│ Average Time (HEAD) │ 355.80ms │
│ Average Time (row-group-morselization) │ 348.88ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch10 — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 40.0s |
| Peak memory | 12.2 GiB |
| Avg memory | 8.5 GiB |
| CPU user | 419.6s |
| CPU sys | 20.0s |
| Peak spill | 0 B |
**tpch10 — branch**
| Metric | Value |
|--------|-------|
| Wall time | 40.0s |
| Peak memory | 11.9 GiB |
| Avg memory | 8.4 GiB |
| CPU user | 417.3s |
| CPU sys | 19.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297698855
@Dandandan can you rebase / merge main to get https://github.com/apache/datafusion/issues/21625 and we can see if that fixes tcph timing out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291938723
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4862301134
Experimenting here: https://github.com/apache/datafusion/pull/23285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296725781
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4296699660-1749-wdn6q 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (52d6bce6e99b9fba1d98d109fa4b22903340c13d) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..52d6bce6e99b9fba1d98d109fa4b22903340c13d) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4296699660
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4298739098
> This looks really cool @Dandandan -- please let me know when it is ready for review (or if you would like help, etc)
I think the perf results look good - I think we mostly need to find out the right API that will also extend to further splitting/merging and other optimizations...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291439513
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291424691-1710-pz7tx 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291026396
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294343524
run benchmark tpch10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291399934
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291385422)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291385422-1709-g6qqr 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: clickbench_1
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291385422
run benchmark clickbench_1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292774996
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292764381-1724-t58bh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed pull request #21766: feat(parquet): row-group morselization for sibling FileStream stealing
URL: https://github.com/apache/datafusion/pull/21766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294316590
🤖 Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4294299393)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4294299393-1741-8bn99 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe)
BENCH_NAME=tpch-10
BENCH_COMMAND=cargo bench --features=parquet --bench tpch-10
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294317135
Benchmark for [this request](https://github.com/apache/datafusion/pull/21766#issuecomment-4294299393) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
struct_query_sql
substr
substr_index
substring
sum
to_char
to_hex
to_local_time
to_time
to_timestamp
topk_aggregate
topk_repartition
translate
trim
trunc
unhex
upper
uuid
window_query_sql
with_hashes
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4294362011
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4294343524)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4294343524-1742-8kg55 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe) using: tpch10
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291435591
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291424691-1711-pg9h8 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..641e6ccdf31a9b163d05006c0c4cfa6f1cc4b13a) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291530610
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291424691)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.68 / 7.22 ±0.93 / 9.08 ms │ 7.18 / 7.61 ±0.76 / 9.14 ms │ 1.05x slower │
│ QQuery 2 │ 145.75 / 147.34 ±1.53 / 149.38 ms │ 143.88 / 144.58 ±0.53 / 145.20 ms │ no change │
│ QQuery 3 │ 112.95 / 114.35 ±1.47 / 116.90 ms │ 112.35 / 112.75 ±0.22 / 112.99 ms │ no change │
│ QQuery 4 │ 1298.74 / 1339.95 ±33.85 / 1376.65 ms │ 1401.20 / 1422.49 ±16.26 / 1449.44 ms │ 1.06x slower │
│ QQuery 5 │ 170.37 / 172.24 ±1.89 / 174.85 ms │ 175.15 / 175.89 ±0.52 / 176.52 ms │ no change │
│ QQuery 6 │ 803.05 / 843.81 ±25.33 / 873.21 ms │ 857.82 / 895.00 ±29.77 / 930.47 ms │ 1.06x slower │
│ QQuery 7 │ 330.15 / 334.83 ±3.92 / 341.87 ms │ 332.58 / 337.52 ±3.91 / 344.13 ms │ no change │
│ QQuery 8 │ 113.44 / 115.60 ±1.72 / 118.62 ms │ 112.97 / 114.57 ±1.21 / 116.70 ms │ no change │
│ QQuery 9 │ 102.49 / 106.04 ±3.17 / 110.16 ms │ 101.33 / 104.03 ±2.24 / 106.63 ms │ no change │
│ QQuery 10 │ 100.96 / 102.93 ±2.21 / 107.23 ms │ 102.48 / 104.95 ±1.57 / 106.81 ms │ no change │
│ QQuery 11 │ 885.15 / 898.08 ±7.45 / 905.36 ms │ 859.93 / 897.35 ±28.17 / 933.22 ms │ no change │
│ QQuery 12 │ 42.40 / 42.72 ±0.32 / 43.11 ms │ 45.13 / 45.62 ±0.58 / 46.72 ms │ 1.07x slower │
│ QQuery 13 │ 388.72 / 393.09 ±4.39 / 400.66 ms │ 386.00 / 390.80 ±4.26 / 396.30 ms │ no change │
│ QQuery 14 │ 974.76 / 983.55 ±7.46 / 993.70 ms │ 974.10 / 981.72 ±5.41 / 990.24 ms │ no change │
│ QQuery 15 │ 14.46 / 15.73 ±2.10 / 19.90 ms │ 15.38 / 15.45 ±0.08 / 15.58 ms │ no change │
│ QQuery 16 │ 7.47 / 7.65 ±0.17 / 7.91 ms │ 7.61 / 7.74 ±0.12 / 7.97 ms │ no change │
│ QQuery 17 │ 222.40 / 226.33 ±3.51 / 231.08 ms │ 218.15 / 224.93 ±5.39 / 232.42 ms │ no change │
│ QQuery 18 │ 123.37 / 125.24 ±1.22 / 126.96 ms │ 122.67 / 126.97 ±2.67 / 129.80 ms │ no change │
│ QQuery 19 │ 152.22 / 153.96 ±1.43 / 156.37 ms │ 153.10 / 157.53 ±5.25 / 165.43 ms │ no change │
│ QQuery 20 │ 12.91 / 13.27 ±0.38 / 13.95 ms │ 13.19 / 13.77 ±0.47 / 14.46 ms │ no change │
│ QQuery 21 │ 18.95 / 19.23 ±0.27 / 19.71 ms │ 18.16 / 18.51 ±0.43 / 19.35 ms │ no change │
│ QQuery 22 │ 472.25 / 488.96 ±11.39 / 502.54 ms │ 474.27 / 489.07 ±13.69 / 506.57 ms │ no change │
│ QQuery 23 │ 826.47 / 843.90 ±22.37 / 887.28 ms │ 818.95 / 836.25 ±10.14 / 847.74 ms │ no change │
│ QQuery 24 │ 385.33 / 393.70 ±4.96 / 398.38 ms │ 369.86 / 377.87 ±7.55 / 391.47 ms │ no change │
│ QQuery 25 │ 342.58 / 345.48 ±1.82 / 348.04 ms │ 329.68 / 331.12 ±1.15 / 333.01 ms │ no change │
│ QQuery 26 │ 77.98 / 78.19 ±0.13 / 78.32 ms │ 77.29 / 77.55 ±0.22 / 77.92 ms │ no change │
│ QQuery 27 │ 7.41 / 7.49 ±0.06 / 7.55 ms │ 6.97 / 7.08 ±0.17 / 7.41 ms │ +1.06x faster │
│ QQuery 28 │ 152.44 / 154.12 ±2.26 / 158.57 ms │ 147.37 / 148.93 ±1.27 / 151.22 ms │ no change │
│ QQuery 29 │ 277.90 / 282.44 ±3.60 / 287.86 ms │ 281.64 / 287.01 ±7.02 / 300.44 ms │ no change │
│ QQuery 30 │ 42.38 / 42.81 ±0.34 / 43.35 ms │ 43.17 / 43.68 ±0.60 / 44.82 ms │ no change │
│ QQuery 31 │ 166.98 / 170.75 ±2.84 / 175.49 ms │ 163.43 / 169.70 ±3.82 / 175.09 ms │ no change │
│ QQuery 32 │ 14.30 / 14.56 ±0.17 / 14.72 ms │ 14.25 / 14.54 ±0.17 / 14.70 ms │ no change │
│ QQuery 33 │ 143.36 / 145.04 ±2.79 / 150.62 ms │ 137.93 / 139.51 ±1.13 / 141.23 ms │ no change │
│ QQuery 34 │ 7.40 / 7.54 ±0.12 / 7.72 ms │ 6.74 / 6.90 ±0.19 / 7.27 ms │ +1.09x faster │
│ QQuery 35 │ 99.50 / 102.87 ±1.97 / 105.14 ms │ 98.84 / 103.36 ±2.71 / 106.31 ms │ no change │
│ QQuery 36 │ 6.42 / 6.71 ±0.20 / 6.94 ms │ 6.72 / 6.87 ±0.10 / 6.98 ms │ no change │
│ QQuery 37 │ 8.06 / 8.14 ±0.06 / 8.23 ms │ 8.04 / 8.18 ±0.09 / 8.30 ms │ no change │
│ QQuery 38 │ 85.59 / 88.64 ±1.93 / 91.53 ms │ 84.71 / 85.73 ±1.36 / 88.41 ms │ no change │
│ QQuery 39 │ 119.88 / 125.54 ±3.99 / 129.04 ms │ 114.05 / 122.82 ±7.99 / 134.17 ms │ no change │
│ QQuery 40 │ 106.12 / 109.32 ±2.09 / 112.57 ms │ 101.06 / 105.98 ±2.94 / 109.89 ms │ no change │
│ QQuery 41 │ 14.69 / 14.86 ±0.15 / 15.06 ms │ 14.89 / 15.12 ±0.29 / 15.67 ms │ no change │
│ QQuery 42 │ 106.89 / 110.18 ±3.26 / 115.77 ms │ 107.20 / 109.21 ±1.70 / 111.63 ms │ no change │
│ QQuery 43 │ 6.04 / 6.10 ±0.07 / 6.22 ms │ 5.57 / 5.71 ±0.18 / 6.06 ms │ +1.07x faster │
│ QQuery 44 │ 12.13 / 12.27 ±0.07 / 12.34 ms │ 11.41 / 11.46 ±0.07 / 11.61 ms │ +1.07x faster │
│ QQuery 45 │ 49.38 / 50.16 ±0.43 / 50.66 ms │ 47.83 / 48.29 ±0.47 / 49.03 ms │ no change │
│ QQuery 46 │ 8.29 / 8.45 ±0.24 / 8.91 ms │ 8.57 / 8.90 ±0.19 / 9.15 ms │ 1.05x slower │
│ QQuery 47 │ 692.02 / 735.34 ±22.23 / 753.79 ms │ 678.74 / 718.41 ±21.63 / 742.40 ms │ no change │
│ QQuery 48 │ 271.01 / 276.73 ±4.79 / 284.87 ms │ 270.73 / 276.72 ±4.73 / 283.42 ms │ no change │
│ QQuery 49 │ 247.84 / 250.22 ±3.02 / 255.32 ms │ 248.77 / 251.86 ±1.70 / 253.39 ms │ no change │
│ QQuery 50 │ 202.83 / 212.48 ±10.20 / 229.42 ms │ 210.91 / 219.67 ±7.71 / 232.85 ms │ no change │
│ QQuery 51 │ 173.84 / 177.13 ±4.05 / 185.04 ms │ 177.98 / 179.15 ±0.62 / 179.65 ms │ no change │
│ QQuery 52 │ 106.27 / 108.91 ±2.08 / 111.93 ms │ 105.45 / 106.39 ±0.79 / 107.27 ms │ no change │
│ QQuery 53 │ 99.96 / 101.80 ±1.87 / 105.41 ms │ 100.57 / 101.95 ±1.34 / 104.32 ms │ no change │
│ QQuery 54 │ 140.75 / 142.25 ±2.08 / 146.31 ms │ 141.12 / 143.78 ±3.05 / 149.56 ms │ no change │
│ QQuery 55 │ 104.11 / 106.65 ±2.10 / 109.02 ms │ 106.15 / 107.32 ±0.84 / 108.49 ms │ no change │
│ QQuery 56 │ 136.31 / 138.98 ±2.25 / 143.00 ms │ 140.50 / 142.09 ±1.29 / 144.41 ms │ no change │
│ QQuery 57 │ 161.60 / 163.91 ±1.75 / 166.45 ms │ 162.02 / 163.59 ±0.88 / 164.53 ms │ no change │
│ QQuery 58 │ 310.26 / 311.78 ±1.26 / 313.63 ms │ 308.81 / 311.82 ±1.73 / 314.23 ms │ no change │
│ QQuery 59 │ 199.36 / 203.59 ±3.12 / 208.99 ms │ 200.26 / 201.20 ±0.77 / 202.42 ms │ no change │
│ QQuery 60 │ 141.95 / 144.64 ±1.65 / 146.73 ms │ 143.62 / 145.44 ±2.19 / 149.73 ms │ no change │
│ QQuery 61 │ 13.44 / 13.65 ±0.19 / 13.89 ms │ 14.09 / 14.25 ±0.14 / 14.48 ms │ no change │
│ QQuery 62 │ 872.42 / 889.02 ±13.68 / 913.73 ms │ 909.23 / 923.07 ±9.75 / 932.99 ms │ no change │
│ QQuery 63 │ 100.98 / 105.68 ±6.05 / 117.25 ms │ 101.23 / 104.50 ±2.59 / 108.80 ms │ no change │
│ QQuery 64 │ 657.37 / 672.04 ±8.22 / 681.82 ms │ 668.03 / 683.85 ±9.65 / 697.40 ms │ no change │
│ QQuery 65 │ 239.62 / 246.30 ±7.96 / 259.98 ms │ 243.73 / 257.53 ±9.39 / 268.07 ms │ no change │
│ QQuery 66 │ 214.55 / 222.70 ±8.29 / 233.06 ms │ 220.70 / 229.54 ±7.41 / 241.38 ms │ no change │
│ QQuery 67 │ 289.85 / 315.84 ±15.20 / 333.62 ms │ 299.83 / 309.09 ±7.24 / 318.89 ms │ no change │
│ QQuery 68 │ 9.04 / 9.20 ±0.14 / 9.44 ms │ 8.49 / 8.84 ±0.26 / 9.28 ms │ no change │
│ QQuery 69 │ 98.77 / 100.46 ±1.76 / 103.82 ms │ 97.78 / 101.32 ±3.27 / 106.69 ms │ no change │
│ QQuery 70 │ 315.84 / 325.51 ±11.24 / 341.09 ms │ 312.81 / 322.89 ±7.08 / 333.74 ms │ no change │
│ QQuery 71 │ 131.74 / 136.04 ±3.58 / 140.79 ms │ 133.64 / 136.06 ±1.68 / 138.41 ms │ no change │
│ QQuery 72 │ 575.83 / 600.35 ±13.52 / 616.59 ms │ 588.41 / 594.33 ±5.52 / 603.30 ms │ no change │
│ QQuery 73 │ 7.10 / 7.27 ±0.10 / 7.41 ms │ 6.51 / 6.76 ±0.21 / 7.08 ms │ +1.08x faster │
│ QQuery 74 │ 530.37 / 562.19 ±35.97 / 615.37 ms │ 532.43 / 571.01 ±26.27 / 607.69 ms │ no change │
│ QQuery 75 │ 264.74 / 272.32 ±4.57 / 276.56 ms │ 264.89 / 266.34 ±1.26 / 268.50 ms │ no change │
│ QQuery 76 │ 128.10 / 130.07 ±1.34 / 131.76 ms │ 130.04 / 132.05 ±1.54 / 133.91 ms │ no change │
│ QQuery 77 │ 184.09 / 187.93 ±3.19 / 193.20 ms │ 185.65 / 188.26 ±2.17 / 191.74 ms │ no change │
│ QQuery 78 │ 329.98 / 338.62 ±6.73 / 347.24 ms │ 336.11 / 339.26 ±2.70 / 343.24 ms │ no change │
│ QQuery 79 │ 227.78 / 243.75 ±8.51 / 253.23 ms │ 247.46 / 252.15 ±3.43 / 256.12 ms │ no change │
│ QQuery 80 │ 316.79 / 320.43 ±3.59 / 326.67 ms │ 319.65 / 324.09 ±3.13 / 329.24 ms │ no change │
│ QQuery 81 │ 26.45 / 26.91 ±0.33 / 27.40 ms │ 26.98 / 28.21 ±1.84 / 31.84 ms │ no change │
│ QQuery 82 │ 39.29 / 39.95 ±0.36 / 40.30 ms │ 38.61 / 39.62 ±0.70 / 40.47 ms │ no change │
│ QQuery 83 │ 36.35 / 36.95 ±0.74 / 38.36 ms │ 36.09 / 36.48 ±0.28 / 36.81 ms │ no change │
│ QQuery 84 │ 45.67 / 45.87 ±0.16 / 46.09 ms │ 46.49 / 47.09 ±0.84 / 48.74 ms │ no change │
│ QQuery 85 │ 139.02 / 139.75 ±1.00 / 141.69 ms │ 141.26 / 145.33 ±4.57 / 154.06 ms │ no change │
│ QQuery 86 │ 37.81 / 38.79 ±1.61 / 42.00 ms │ 37.08 / 37.40 ±0.41 / 38.20 ms │ no change │
│ QQuery 87 │ 3.75 / 3.89 ±0.15 / 4.18 ms │ 3.42 / 3.52 ±0.13 / 3.78 ms │ +1.10x faster │
│ QQuery 88 │ 101.09 / 102.08 ±1.17 / 104.22 ms │ 98.09 / 101.38 ±2.63 / 105.25 ms │ no change │
│ QQuery 89 │ 117.84 / 119.48 ±2.29 / 123.96 ms │ 114.44 / 114.96 ±0.35 / 115.30 ms │ no change │
│ QQuery 90 │ 21.97 / 22.64 ±0.57 / 23.47 ms │ 22.67 / 23.33 ±0.44 / 24.04 ms │ no change │
│ QQuery 91 │ 57.15 / 57.79 ±0.45 / 58.55 ms │ 59.37 / 61.31 ±1.06 / 62.31 ms │ 1.06x slower │
│ QQuery 92 │ 55.55 / 55.86 ±0.40 / 56.61 ms │ 55.39 / 56.22 ±0.65 / 57.31 ms │ no change │
│ QQuery 93 │ 184.38 / 187.62 ±2.16 / 189.93 ms │ 179.24 / 183.28 ±4.69 / 192.38 ms │ no change │
│ QQuery 94 │ 62.04 / 62.50 ±0.55 / 63.45 ms │ 59.18 / 59.67 ±0.32 / 60.04 ms │ no change │
│ QQuery 95 │ 127.87 / 128.49 ±0.59 / 129.56 ms │ 124.86 / 126.30 ±1.28 / 128.56 ms │ no change │
│ QQuery 96 │ 69.78 / 72.38 ±2.82 / 77.56 ms │ 67.32 / 68.66 ±1.05 / 70.03 ms │ +1.05x faster │
│ QQuery 97 │ 121.63 / 124.29 ±1.40 / 125.47 ms │ 116.20 / 119.76 ±3.58 / 125.26 ms │ no change │
│ QQuery 98 │ 157.67 / 159.61 ±1.70 / 162.21 ms │ 147.51 / 151.37 ±3.16 / 155.65 ms │ +1.05x faster │
│ QQuery 99 │ 10828.37 / 10886.49 ±35.01 / 10927.45 ms │ 10878.55 / 11136.01 ±231.41 / 11553.51 ms │ no change │
└───────────┴──────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 30866.53ms │
│ Total Time (row-group-morselization) │ 31235.08ms │
│ Average Time (HEAD) │ 311.78ms │
│ Average Time (row-group-morselization) │ 315.51ms │
│ Queries Faster │ 8 │
│ Queries Slower │ 6 │
│ Queries with No Change │ 85 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.3 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 258.6s |
| CPU sys | 8.2s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 259.8s |
| CPU sys | 8.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291700146
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291595788)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.71 / 7.29 ±0.88 / 9.04 ms │ 6.86 / 7.32 ±0.87 / 9.07 ms │ no change │
│ QQuery 2 │ 146.05 / 146.57 ±0.46 / 147.31 ms │ 142.83 / 143.37 ±0.28 / 143.61 ms │ no change │
│ QQuery 3 │ 111.87 / 112.60 ±1.16 / 114.89 ms │ 111.47 / 112.07 ±0.51 / 112.82 ms │ no change │
│ QQuery 4 │ 1210.66 / 1215.88 ±4.24 / 1223.21 ms │ 1204.73 / 1210.73 ±5.26 / 1218.34 ms │ no change │
│ QQuery 5 │ 170.19 / 172.07 ±1.58 / 174.07 ms │ 171.48 / 172.91 ±0.98 / 174.07 ms │ no change │
│ QQuery 6 │ 810.50 / 818.15 ±6.57 / 826.47 ms │ 814.69 / 825.14 ±9.74 / 840.38 ms │ no change │
│ QQuery 7 │ 332.35 / 333.39 ±1.12 / 335.41 ms │ 332.91 / 335.33 ±1.47 / 337.36 ms │ no change │
│ QQuery 8 │ 112.57 / 112.90 ±0.32 / 113.32 ms │ 111.73 / 112.51 ±0.77 / 113.62 ms │ no change │
│ QQuery 9 │ 100.30 / 103.47 ±2.58 / 106.14 ms │ 100.08 / 107.30 ±5.03 / 115.03 ms │ no change │
│ QQuery 10 │ 100.75 / 101.37 ±0.45 / 102.11 ms │ 100.45 / 100.83 ±0.38 / 101.53 ms │ no change │
│ QQuery 11 │ 821.71 / 829.47 ±6.72 / 838.67 ms │ 808.70 / 815.48 ±4.88 / 820.41 ms │ no change │
│ QQuery 12 │ 42.47 / 42.74 ±0.26 / 43.09 ms │ 42.51 / 43.06 ±0.55 / 44.06 ms │ no change │
│ QQuery 13 │ 385.32 / 386.53 ±0.97 / 388.06 ms │ 385.54 / 388.17 ±1.91 / 390.62 ms │ no change │
│ QQuery 14 │ 976.11 / 981.17 ±3.37 / 986.06 ms │ 971.44 / 979.42 ±4.70 / 985.12 ms │ no change │
│ QQuery 15 │ 14.15 / 14.44 ±0.26 / 14.87 ms │ 14.13 / 14.45 ±0.30 / 14.97 ms │ no change │
│ QQuery 16 │ 7.27 / 7.35 ±0.11 / 7.57 ms │ 7.26 / 7.37 ±0.11 / 7.56 ms │ no change │
│ QQuery 17 │ 221.18 / 222.30 ±1.70 / 225.67 ms │ 219.91 / 221.65 ±2.03 / 225.53 ms │ no change │
│ QQuery 18 │ 121.16 / 122.39 ±1.21 / 124.62 ms │ 121.46 / 122.44 ±1.03 / 124.36 ms │ no change │
│ QQuery 19 │ 151.66 / 152.86 ±0.69 / 153.64 ms │ 151.89 / 154.05 ±1.46 / 156.17 ms │ no change │
│ QQuery 20 │ 12.67 / 12.94 ±0.25 / 13.38 ms │ 12.99 / 13.16 ±0.12 / 13.31 ms │ no change │
│ QQuery 21 │ 18.59 / 18.90 ±0.24 / 19.28 ms │ 18.65 / 18.90 ±0.17 / 19.09 ms │ no change │
│ QQuery 22 │ 470.97 / 474.33 ±2.22 / 476.62 ms │ 477.59 / 483.73 ±3.33 / 487.13 ms │ no change │
│ QQuery 23 │ 804.01 / 807.53 ±3.69 / 813.67 ms │ 800.03 / 807.35 ±4.26 / 812.18 ms │ no change │
│ QQuery 24 │ 371.02 / 373.99 ±2.73 / 378.20 ms │ 370.52 / 374.70 ±3.16 / 379.46 ms │ no change │
│ QQuery 25 │ 331.08 / 332.94 ±1.27 / 334.96 ms │ 330.31 / 333.01 ±1.56 / 334.87 ms │ no change │
│ QQuery 26 │ 75.66 / 76.53 ±0.76 / 77.87 ms │ 76.48 / 76.72 ±0.24 / 77.14 ms │ no change │
│ QQuery 27 │ 6.89 / 6.98 ±0.10 / 7.15 ms │ 6.81 / 6.88 ±0.09 / 7.05 ms │ no change │
│ QQuery 28 │ 146.70 / 147.61 ±0.76 / 149.01 ms │ 147.56 / 157.96 ±17.52 / 192.65 ms │ 1.07x slower │
│ QQuery 29 │ 269.26 / 272.46 ±3.33 / 278.90 ms │ 270.24 / 272.94 ±2.04 / 276.36 ms │ no change │
│ QQuery 30 │ 40.66 / 41.73 ±1.61 / 44.92 ms │ 40.92 / 42.55 ±2.39 / 47.29 ms │ no change │
│ QQuery 31 │ 162.25 / 163.89 ±1.11 / 165.47 ms │ 163.30 / 166.56 ±3.22 / 172.25 ms │ no change │
│ QQuery 32 │ 13.26 / 14.59 ±2.06 / 18.70 ms │ 13.04 / 13.60 ±0.49 / 14.50 ms │ +1.07x faster │
│ QQuery 33 │ 137.17 / 137.66 ±0.61 / 138.87 ms │ 137.32 / 138.19 ±0.76 / 139.51 ms │ no change │
│ QQuery 34 │ 6.81 / 6.92 ±0.15 / 7.22 ms │ 6.93 / 7.06 ±0.16 / 7.34 ms │ no change │
│ QQuery 35 │ 98.67 / 100.56 ±1.34 / 102.40 ms │ 99.53 / 100.45 ±0.49 / 100.90 ms │ no change │
│ QQuery 36 │ 6.43 / 6.70 ±0.29 / 7.21 ms │ 6.57 / 6.85 ±0.26 / 7.27 ms │ no change │
│ QQuery 37 │ 8.06 / 8.12 ±0.04 / 8.18 ms │ 8.12 / 8.23 ±0.10 / 8.38 ms │ no change │
│ QQuery 38 │ 84.05 / 85.75 ±1.48 / 87.72 ms │ 86.24 / 86.91 ±0.77 / 88.40 ms │ no change │
│ QQuery 39 │ 117.95 / 119.36 ±2.54 / 124.44 ms │ 117.11 / 120.12 ±3.45 / 126.74 ms │ no change │
│ QQuery 40 │ 101.81 / 105.34 ±3.45 / 110.93 ms │ 101.92 / 107.71 ±3.11 / 110.29 ms │ no change │
│ QQuery 41 │ 13.96 / 14.12 ±0.19 / 14.50 ms │ 14.00 / 14.88 ±1.43 / 17.70 ms │ 1.05x slower │
│ QQuery 42 │ 104.80 / 107.42 ±2.63 / 112.28 ms │ 105.86 / 108.01 ±3.52 / 115.03 ms │ no change │
│ QQuery 43 │ 5.74 / 5.83 ±0.13 / 6.07 ms │ 5.80 / 5.90 ±0.12 / 6.13 ms │ no change │
│ QQuery 44 │ 11.71 / 11.91 ±0.18 / 12.23 ms │ 11.67 / 11.81 ±0.12 / 12.03 ms │ no change │
│ QQuery 45 │ 47.86 / 48.29 ±0.30 / 48.71 ms │ 48.31 / 48.58 ±0.21 / 48.87 ms │ no change │
│ QQuery 46 │ 8.28 / 8.50 ±0.22 / 8.93 ms │ 8.36 / 8.54 ±0.19 / 8.92 ms │ no change │
│ QQuery 47 │ 656.83 / 664.80 ±5.34 / 670.87 ms │ 664.72 / 669.91 ±4.67 / 675.89 ms │ no change │
│ QQuery 48 │ 270.95 / 273.13 ±1.87 / 276.50 ms │ 270.13 / 273.97 ±3.56 / 279.26 ms │ no change │
│ QQuery 49 │ 247.19 / 248.69 ±0.98 / 249.74 ms │ 248.12 / 249.99 ±2.20 / 253.65 ms │ no change │
│ QQuery 50 │ 199.94 / 205.36 ±6.08 / 216.59 ms │ 202.23 / 207.90 ±4.55 / 213.81 ms │ no change │
│ QQuery 51 │ 174.68 / 177.56 ±3.30 / 183.58 ms │ 175.91 / 178.21 ±3.28 / 184.66 ms │ no change │
│ QQuery 52 │ 105.20 / 106.98 ±2.22 / 111.34 ms │ 105.47 / 107.44 ±1.81 / 110.81 ms │ no change │
│ QQuery 53 │ 99.45 / 100.70 ±0.81 / 101.96 ms │ 100.18 / 100.52 ±0.36 / 100.99 ms │ no change │
│ QQuery 54 │ 141.04 / 143.12 ±3.16 / 149.40 ms │ 142.34 / 143.30 ±1.30 / 145.80 ms │ no change │
│ QQuery 55 │ 103.88 / 104.45 ±0.44 / 105.08 ms │ 104.92 / 106.11 ±1.04 / 107.94 ms │ no change │
│ QQuery 56 │ 137.88 / 138.98 ±1.13 / 140.71 ms │ 137.43 / 138.17 ±0.88 / 139.86 ms │ no change │
│ QQuery 57 │ 163.66 / 165.58 ±2.20 / 169.82 ms │ 162.67 / 163.95 ±1.05 / 165.12 ms │ no change │
│ QQuery 58 │ 306.56 / 308.29 ±2.56 / 313.37 ms │ 305.65 / 307.87 ±1.89 / 310.46 ms │ no change │
│ QQuery 59 │ 194.61 / 196.86 ±1.69 / 199.15 ms │ 191.47 / 193.28 ±1.82 / 195.51 ms │ no change │
│ QQuery 60 │ 138.60 / 140.68 ±2.02 / 144.11 ms │ 139.10 / 140.40 ±0.94 / 141.70 ms │ no change │
│ QQuery 61 │ 13.12 / 13.30 ±0.20 / 13.70 ms │ 13.31 / 13.44 ±0.14 / 13.70 ms │ no change │
│ QQuery 62 │ 853.40 / 861.32 ±6.59 / 869.77 ms │ 847.06 / 856.55 ±6.37 / 865.01 ms │ no change │
│ QQuery 63 │ 100.24 / 103.52 ±4.12 / 111.55 ms │ 100.42 / 102.82 ±3.09 / 108.13 ms │ no change │
│ QQuery 64 │ 648.46 / 655.24 ±4.01 / 659.77 ms │ 651.91 / 656.80 ±4.07 / 661.72 ms │ no change │
│ QQuery 65 │ 237.83 / 240.98 ±2.70 / 245.19 ms │ 237.54 / 240.68 ±4.28 / 249.08 ms │ no change │
│ QQuery 66 │ 211.93 / 219.29 ±7.44 / 232.55 ms │ 210.35 / 217.76 ±7.37 / 228.01 ms │ no change │
│ QQuery 67 │ 286.88 / 291.33 ±4.27 / 297.70 ms │ 291.16 / 304.03 ±14.13 / 329.61 ms │ no change │
│ QQuery 68 │ 8.28 / 8.51 ±0.22 / 8.92 ms │ 8.46 / 8.69 ±0.26 / 9.19 ms │ no change │
│ QQuery 69 │ 96.06 / 97.11 ±0.65 / 98.08 ms │ 96.42 / 97.40 ±1.13 / 99.41 ms │ no change │
│ QQuery 70 │ 310.95 / 318.03 ±4.37 / 323.27 ms │ 312.70 / 324.33 ±7.80 / 331.44 ms │ no change │
│ QQuery 71 │ 130.27 / 134.34 ±4.53 / 142.25 ms │ 131.73 / 134.56 ±3.09 / 140.30 ms │ no change │
│ QQuery 72 │ 575.79 / 581.50 ±3.04 / 584.55 ms │ 566.24 / 579.59 ±7.35 / 586.36 ms │ no change │
│ QQuery 73 │ 6.58 / 6.75 ±0.23 / 7.20 ms │ 6.61 / 6.74 ±0.22 / 7.19 ms │ no change │
│ QQuery 74 │ 513.22 / 519.04 ±4.05 / 522.59 ms │ 514.61 / 518.31 ±2.38 / 520.98 ms │ no change │
│ QQuery 75 │ 266.25 / 267.11 ±0.86 / 268.61 ms │ 265.39 / 268.73 ±2.46 / 271.94 ms │ no change │
│ QQuery 76 │ 128.06 / 129.24 ±1.57 / 132.33 ms │ 127.58 / 129.36 ±1.15 / 130.95 ms │ no change │
│ QQuery 77 │ 185.15 / 185.62 ±0.48 / 186.47 ms │ 184.95 / 187.78 ±3.56 / 194.77 ms │ no change │
│ QQuery 78 │ 324.49 / 328.28 ±3.06 / 331.49 ms │ 325.31 / 326.96 ±1.35 / 329.04 ms │ no change │
│ QQuery 79 │ 226.61 / 227.72 ±1.09 / 229.45 ms │ 224.78 / 227.34 ±1.42 / 228.86 ms │ no change │
│ QQuery 80 │ 314.69 / 317.19 ±2.05 / 319.97 ms │ 316.04 / 320.37 ±2.73 / 323.66 ms │ no change │
│ QQuery 81 │ 25.34 / 25.78 ±0.40 / 26.28 ms │ 25.23 / 25.61 ±0.21 / 25.77 ms │ no change │
│ QQuery 82 │ 38.59 / 39.00 ±0.37 / 39.58 ms │ 38.48 / 38.71 ±0.16 / 38.96 ms │ no change │
│ QQuery 83 │ 36.35 / 36.58 ±0.15 / 36.74 ms │ 36.07 / 36.71 ±0.48 / 37.46 ms │ no change │
│ QQuery 84 │ 44.95 / 45.31 ±0.30 / 45.82 ms │ 45.42 / 45.86 ±0.33 / 46.38 ms │ no change │
│ QQuery 85 │ 138.02 / 139.20 ±0.85 / 140.25 ms │ 138.60 / 140.40 ±1.51 / 142.73 ms │ no change │
│ QQuery 86 │ 36.34 / 36.68 ±0.29 / 37.13 ms │ 36.31 / 36.84 ±0.34 / 37.34 ms │ no change │
│ QQuery 87 │ 3.49 / 3.59 ±0.13 / 3.84 ms │ 3.43 / 3.52 ±0.13 / 3.77 ms │ no change │
│ QQuery 88 │ 97.68 / 99.31 ±1.30 / 101.25 ms │ 98.58 / 101.22 ±4.56 / 110.32 ms │ no change │
│ QQuery 89 │ 114.42 / 115.20 ±0.51 / 115.85 ms │ 114.51 / 115.28 ±0.51 / 116.04 ms │ no change │
│ QQuery 90 │ 21.85 / 22.48 ±0.46 / 23.22 ms │ 21.73 / 22.04 ±0.27 / 22.49 ms │ no change │
│ QQuery 91 │ 57.11 / 58.40 ±1.50 / 61.22 ms │ 58.27 / 60.28 ±2.78 / 65.69 ms │ no change │
│ QQuery 92 │ 55.15 / 55.70 ±0.39 / 56.13 ms │ 55.37 / 55.88 ±0.46 / 56.57 ms │ no change │
│ QQuery 93 │ 178.34 / 179.90 ±1.05 / 181.21 ms │ 179.02 / 179.78 ±0.72 / 180.85 ms │ no change │
│ QQuery 94 │ 59.46 / 59.80 ±0.40 / 60.38 ms │ 59.28 / 59.98 ±0.63 / 60.92 ms │ no change │
│ QQuery 95 │ 124.38 / 125.21 ±0.75 / 126.53 ms │ 124.15 / 125.99 ±1.95 / 129.54 ms │ no change │
│ QQuery 96 │ 66.81 / 68.35 ±1.22 / 70.51 ms │ 66.70 / 68.57 ±1.10 / 69.97 ms │ no change │
│ QQuery 97 │ 114.18 / 116.11 ±1.84 / 119.57 ms │ 116.15 / 117.22 ±0.60 / 117.76 ms │ no change │
│ QQuery 98 │ 147.42 / 148.44 ±0.91 / 150.08 ms │ 149.19 / 150.44 ±0.97 / 151.85 ms │ no change │
│ QQuery 99 │ 10671.03 / 10871.11 ±124.85 / 11042.79 ms │ 10673.96 / 10825.91 ±105.96 / 10972.86 ms │ no change │
└───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 30120.54ms │
│ Total Time (row-group-morselization) │ 30128.41ms │
│ Average Time (HEAD) │ 304.25ms │
│ Average Time (row-group-morselization) │ 304.33ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 96 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.5 GiB |
| Avg memory | 5.6 GiB |
| CPU user | 251.2s |
| CPU sys | 8.4s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 251.3s |
| CPU sys | 8.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292774542
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292764381-1722-zl8hz 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292774851
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292764381-1723-xd4zj 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (26f09e43363279eb519d74d106f4ca6ed13b7dbe) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..26f09e43363279eb519d74d106f4ca6ed13b7dbe) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4292829277
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4292764381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 7.29 / 7.74 ±0.65 / 9.04 ms │ 7.30 / 7.80 ±0.71 / 9.20 ms │ no change │
│ QQuery 2 │ 149.72 / 150.98 ±1.12 / 153.07 ms │ 146.01 / 146.47 ±0.47 / 147.34 ms │ no change │
│ QQuery 3 │ 114.35 / 115.27 ±0.60 / 116.02 ms │ 114.69 / 115.86 ±0.71 / 116.62 ms │ no change │
│ QQuery 4 │ 1431.96 / 1437.05 ±4.45 / 1443.18 ms │ 1433.83 / 1443.29 ±9.09 / 1457.92 ms │ no change │
│ QQuery 5 │ 175.67 / 177.05 ±1.21 / 179.25 ms │ 175.24 / 177.57 ±1.72 / 180.30 ms │ no change │
│ QQuery 6 │ 870.09 / 886.77 ±13.14 / 909.50 ms │ 863.94 / 894.30 ±22.17 / 921.09 ms │ no change │
│ QQuery 7 │ 339.35 / 343.85 ±3.51 / 348.58 ms │ 345.37 / 346.85 ±1.31 / 348.88 ms │ no change │
│ QQuery 8 │ 115.22 / 115.96 ±0.70 / 117.27 ms │ 114.85 / 116.15 ±1.17 / 117.76 ms │ no change │
│ QQuery 9 │ 101.81 / 105.65 ±3.01 / 108.71 ms │ 101.69 / 105.68 ±3.24 / 109.14 ms │ no change │
│ QQuery 10 │ 104.44 / 105.08 ±0.40 / 105.60 ms │ 104.29 / 105.29 ±0.72 / 106.15 ms │ no change │
│ QQuery 11 │ 1006.37 / 1023.16 ±10.09 / 1035.90 ms │ 1022.12 / 1033.20 ±7.32 / 1044.18 ms │ no change │
│ QQuery 12 │ 45.47 / 45.92 ±0.35 / 46.41 ms │ 46.66 / 46.86 ±0.13 / 47.07 ms │ no change │
│ QQuery 13 │ 406.41 / 408.72 ±1.49 / 410.28 ms │ 406.77 / 413.17 ±4.68 / 418.53 ms │ no change │
│ QQuery 14 │ 998.78 / 1008.01 ±6.61 / 1017.36 ms │ 996.09 / 1002.34 ±4.72 / 1009.63 ms │ no change │
│ QQuery 15 │ 15.80 / 16.09 ±0.20 / 16.39 ms │ 15.70 / 15.84 ±0.17 / 16.13 ms │ no change │
│ QQuery 16 │ 8.01 / 8.14 ±0.11 / 8.30 ms │ 7.91 / 8.02 ±0.15 / 8.32 ms │ no change │
│ QQuery 17 │ 232.66 / 234.42 ±0.98 / 235.37 ms │ 231.74 / 234.61 ±2.10 / 237.58 ms │ no change │
│ QQuery 18 │ 127.13 / 128.18 ±0.87 / 129.63 ms │ 126.86 / 128.13 ±1.38 / 129.83 ms │ no change │
│ QQuery 19 │ 163.30 / 164.62 ±0.85 / 165.95 ms │ 163.38 / 165.39 ±1.77 / 167.47 ms │ no change │
│ QQuery 20 │ 13.66 / 14.03 ±0.31 / 14.59 ms │ 13.90 / 14.16 ±0.14 / 14.33 ms │ no change │
│ QQuery 21 │ 19.82 / 20.32 ±0.43 / 20.93 ms │ 20.04 / 20.47 ±0.30 / 20.86 ms │ no change │
│ QQuery 22 │ 510.46 / 522.13 ±8.88 / 537.31 ms │ 545.94 / 555.59 ±6.59 / 563.77 ms │ 1.06x slower │
│ QQuery 23 │ 896.76 / 900.77 ±3.00 / 906.04 ms │ 901.18 / 912.74 ±10.80 / 932.29 ms │ no change │
│ QQuery 24 │ 391.87 / 395.81 ±3.86 / 402.95 ms │ 396.11 / 399.72 ±3.03 / 404.46 ms │ no change │
│ QQuery 25 │ 345.88 / 348.90 ±2.24 / 351.38 ms │ 346.90 / 351.03 ±3.80 / 356.70 ms │ no change │
│ QQuery 26 │ 78.75 / 79.66 ±0.58 / 80.49 ms │ 79.72 / 81.40 ±3.07 / 87.53 ms │ no change │
│ QQuery 27 │ 7.28 / 7.46 ±0.13 / 7.59 ms │ 7.41 / 7.58 ±0.09 / 7.66 ms │ no change │
│ QQuery 28 │ 152.69 / 154.33 ±1.33 / 156.65 ms │ 152.70 / 153.64 ±1.51 / 156.65 ms │ no change │
│ QQuery 29 │ 283.95 / 288.09 ±3.52 / 294.53 ms │ 282.72 / 287.35 ±4.86 / 296.33 ms │ no change │
│ QQuery 30 │ 42.73 / 44.11 ±1.69 / 47.41 ms │ 42.43 / 43.00 ±0.45 / 43.69 ms │ no change │
│ QQuery 31 │ 169.70 / 172.77 ±3.31 / 179.07 ms │ 170.12 / 173.03 ±1.72 / 175.12 ms │ no change │
│ QQuery 32 │ 14.62 / 14.90 ±0.17 / 15.15 ms │ 14.47 / 14.80 ±0.30 / 15.25 ms │ no change │
│ QQuery 33 │ 143.92 / 146.02 ±1.16 / 147.28 ms │ 144.83 / 147.31 ±1.98 / 150.90 ms │ no change │
│ QQuery 34 │ 7.55 / 7.69 ±0.11 / 7.83 ms │ 7.64 / 7.76 ±0.09 / 7.86 ms │ no change │
│ QQuery 35 │ 105.19 / 106.57 ±1.95 / 110.45 ms │ 101.91 / 104.58 ±1.83 / 106.94 ms │ no change │
│ QQuery 36 │ 6.97 / 7.10 ±0.08 / 7.18 ms │ 7.04 / 7.34 ±0.20 / 7.66 ms │ no change │
│ QQuery 37 │ 8.82 / 8.94 ±0.10 / 9.09 ms │ 8.80 / 8.87 ±0.04 / 8.93 ms │ no change │
│ QQuery 38 │ 92.38 / 94.38 ±1.99 / 96.85 ms │ 91.72 / 96.45 ±6.07 / 108.39 ms │ no change │
│ QQuery 39 │ 130.37 / 135.59 ±2.71 / 138.04 ms │ 132.74 / 134.70 ±1.06 / 135.71 ms │ no change │
│ QQuery 40 │ 107.14 / 112.61 ±4.03 / 117.58 ms │ 108.81 / 111.70 ±2.50 / 115.33 ms │ no change │
│ QQuery 41 │ 14.87 / 15.09 ±0.16 / 15.33 ms │ 14.71 / 14.92 ±0.21 / 15.31 ms │ no change │
│ QQuery 42 │ 108.74 / 110.22 ±2.33 / 114.86 ms │ 108.70 / 109.38 ±0.39 / 109.89 ms │ no change │
│ QQuery 43 │ 6.10 / 6.15 ±0.04 / 6.22 ms │ 6.04 / 6.11 ±0.10 / 6.30 ms │ no change │
│ QQuery 44 │ 12.41 / 12.54 ±0.10 / 12.69 ms │ 12.43 / 12.59 ±0.17 / 12.88 ms │ no change │
│ QQuery 45 │ 50.37 / 51.17 ±0.85 / 52.74 ms │ 50.62 / 50.93 ±0.32 / 51.55 ms │ no change │
│ QQuery 46 │ 9.08 / 9.36 ±0.21 / 9.71 ms │ 8.88 / 9.11 ±0.20 / 9.37 ms │ no change │
│ QQuery 47 │ 815.24 / 825.10 ±6.23 / 831.00 ms │ 830.10 / 838.33 ±7.29 / 851.83 ms │ no change │
│ QQuery 48 │ 282.83 / 286.78 ±2.56 / 290.41 ms │ 286.62 / 291.14 ±4.64 / 298.93 ms │ no change │
│ QQuery 49 │ 254.36 / 255.52 ±0.92 / 256.86 ms │ 253.31 / 255.69 ±2.45 / 259.37 ms │ no change │
│ QQuery 50 │ 218.88 / 227.19 ±4.50 / 232.48 ms │ 220.15 / 225.40 ±4.04 / 231.69 ms │ no change │
│ QQuery 51 │ 181.22 / 183.68 ±2.10 / 186.45 ms │ 180.60 / 182.76 ±1.58 / 184.87 ms │ no change │
│ QQuery 52 │ 107.39 / 108.40 ±0.87 / 109.86 ms │ 109.30 / 109.51 ±0.22 / 109.91 ms │ no change │
│ QQuery 53 │ 104.42 / 106.10 ±1.49 / 108.66 ms │ 104.15 / 105.66 ±1.50 / 108.51 ms │ no change │
│ QQuery 54 │ 149.30 / 151.26 ±1.40 / 153.65 ms │ 150.62 / 153.27 ±2.80 / 158.62 ms │ no change │
│ QQuery 55 │ 106.76 / 108.09 ±0.95 / 109.61 ms │ 106.97 / 107.99 ±0.91 / 109.19 ms │ no change │
│ QQuery 56 │ 143.88 / 147.24 ±2.22 / 150.88 ms │ 144.80 / 145.76 ±0.88 / 147.20 ms │ no change │
│ QQuery 57 │ 172.54 / 174.47 ±1.25 / 176.46 ms │ 174.83 / 175.43 ±0.46 / 176.12 ms │ no change │
│ QQuery 58 │ 315.75 / 317.51 ±1.50 / 319.74 ms │ 316.15 / 317.39 ±1.30 / 319.61 ms │ no change │
│ QQuery 59 │ 209.41 / 211.30 ±1.65 / 213.21 ms │ 205.54 / 207.84 ±2.75 / 213.21 ms │ no change │
│ QQuery 60 │ 145.79 / 147.56 ±1.28 / 149.21 ms │ 145.89 / 147.66 ±1.12 / 149.18 ms │ no change │
│ QQuery 61 │ 14.04 / 14.22 ±0.10 / 14.33 ms │ 14.48 / 14.56 ±0.09 / 14.73 ms │ no change │
│ QQuery 62 │ 915.71 / 925.94 ±9.58 / 943.25 ms │ 927.69 / 931.75 ±4.09 / 939.33 ms │ no change │
│ QQuery 63 │ 104.02 / 105.58 ±2.03 / 109.54 ms │ 106.07 / 106.85 ±0.50 / 107.64 ms │ no change │
│ QQuery 64 │ 699.32 / 703.06 ±2.69 / 706.37 ms │ 707.00 / 713.09 ±9.21 / 730.88 ms │ no change │
│ QQuery 65 │ 270.85 / 274.77 ±4.38 / 282.22 ms │ 274.84 / 277.63 ±3.28 / 283.84 ms │ no change │
│ QQuery 66 │ 221.84 / 232.29 ±10.24 / 246.22 ms │ 222.47 / 236.80 ±13.23 / 253.67 ms │ no change │
│ QQuery 67 │ 313.74 / 325.03 ±9.17 / 338.02 ms │ 320.11 / 329.93 ±7.55 / 339.05 ms │ no change │
│ QQuery 68 │ 9.23 / 9.47 ±0.14 / 9.63 ms │ 9.36 / 9.60 ±0.15 / 9.74 ms │ no change │
│ QQuery 69 │ 101.05 / 102.04 ±1.05 / 103.66 ms │ 100.35 / 101.96 ±1.34 / 103.80 ms │ no change │
│ QQuery 70 │ 319.39 / 336.90 ±9.27 / 346.89 ms │ 323.69 / 336.80 ±13.53 / 360.29 ms │ no change │
│ QQuery 71 │ 137.58 / 138.91 ±1.84 / 142.38 ms │ 138.22 / 140.29 ±1.90 / 143.59 ms │ no change │
│ QQuery 72 │ 625.03 / 633.67 ±8.48 / 645.30 ms │ 622.13 / 635.11 ±7.59 / 643.52 ms │ no change │
│ QQuery 73 │ 7.14 / 7.25 ±0.08 / 7.37 ms │ 7.33 / 7.51 ±0.11 / 7.64 ms │ no change │
│ QQuery 74 │ 655.44 / 659.96 ±4.36 / 668.01 ms │ 667.26 / 673.12 ±4.47 / 680.40 ms │ no change │
│ QQuery 75 │ 274.25 / 276.77 ±1.60 / 279.26 ms │ 274.83 / 275.71 ±0.62 / 276.67 ms │ no change │
│ QQuery 76 │ 135.59 / 138.02 ±2.55 / 142.69 ms │ 133.93 / 136.67 ±1.83 / 138.77 ms │ no change │
│ QQuery 77 │ 189.24 / 191.35 ±1.42 / 193.13 ms │ 190.35 / 191.40 ±0.87 / 192.69 ms │ no change │
│ QQuery 78 │ 342.53 / 346.83 ±3.60 / 350.86 ms │ 343.98 / 349.11 ±3.17 / 353.25 ms │ no change │
│ QQuery 79 │ 252.06 / 254.53 ±2.64 / 258.60 ms │ 254.03 / 259.57 ±3.25 / 263.00 ms │ no change │
│ QQuery 80 │ 322.50 / 324.35 ±1.95 / 327.22 ms │ 325.82 / 327.66 ±2.02 / 331.47 ms │ no change │
│ QQuery 81 │ 26.54 / 27.02 ±0.26 / 27.27 ms │ 27.12 / 27.75 ±0.53 / 28.65 ms │ no change │
│ QQuery 82 │ 40.01 / 40.35 ±0.25 / 40.73 ms │ 41.00 / 41.51 ±0.43 / 42.20 ms │ no change │
│ QQuery 83 │ 37.85 / 38.42 ±0.46 / 39.13 ms │ 39.06 / 39.56 ±0.26 / 39.78 ms │ no change │
│ QQuery 84 │ 47.03 / 47.27 ±0.21 / 47.65 ms │ 47.91 / 48.31 ±0.29 / 48.70 ms │ no change │
│ QQuery 85 │ 143.33 / 144.90 ±1.76 / 148.14 ms │ 145.07 / 146.27 ±1.37 / 148.84 ms │ no change │
│ QQuery 86 │ 37.91 / 38.01 ±0.09 / 38.12 ms │ 38.80 / 39.36 ±0.47 / 40.07 ms │ no change │
│ QQuery 87 │ 3.71 / 3.84 ±0.15 / 4.07 ms │ 3.71 / 3.83 ±0.10 / 4.01 ms │ no change │
│ QQuery 88 │ 101.76 / 104.10 ±1.60 / 106.33 ms │ 103.39 / 104.77 ±1.69 / 108.01 ms │ no change │
│ QQuery 89 │ 117.61 / 118.90 ±1.02 / 120.53 ms │ 120.71 / 122.72 ±2.32 / 127.23 ms │ no change │
│ QQuery 90 │ 23.25 / 25.03 ±2.88 / 30.76 ms │ 23.30 / 23.37 ±0.08 / 23.52 ms │ +1.07x faster │
│ QQuery 91 │ 59.30 / 60.01 ±0.59 / 61.11 ms │ 60.53 / 61.06 ±0.45 / 61.62 ms │ no change │
│ QQuery 92 │ 57.40 / 58.01 ±0.40 / 58.67 ms │ 58.36 / 59.16 ±1.00 / 61.11 ms │ no change │
│ QQuery 93 │ 190.26 / 191.33 ±0.92 / 192.61 ms │ 190.27 / 192.63 ±1.83 / 195.00 ms │ no change │
│ QQuery 94 │ 61.88 / 62.26 ±0.25 / 62.56 ms │ 62.63 / 63.42 ±0.45 / 63.89 ms │ no change │
│ QQuery 95 │ 128.09 / 128.95 ±0.44 / 129.28 ms │ 129.54 / 130.58 ±1.21 / 132.91 ms │ no change │
│ QQuery 96 │ 69.91 / 71.37 ±1.70 / 74.26 ms │ 70.61 / 73.18 ±1.37 / 74.61 ms │ no change │
│ QQuery 97 │ 123.62 / 125.99 ±2.31 / 130.32 ms │ 127.09 / 129.34 ±1.55 / 131.86 ms │ no change │
│ QQuery 98 │ 158.61 / 161.00 ±1.59 / 163.41 ms │ 162.26 / 163.33 ±0.63 / 164.05 ms │ no change │
│ QQuery 99 │ 10915.36 / 10939.16 ±40.08 / 11019.13 ms │ 10956.65 / 11012.98 ±40.82 / 11080.90 ms │ no change │
└───────────┴──────────────────────────────────────────┴──────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 31906.36ms │
│ Total Time (row-group-morselization) │ 32145.17ms │
│ Average Time (HEAD) │ 322.29ms │
│ Average Time (row-group-morselization) │ 324.70ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 97 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 268.4s |
| CPU sys | 8.3s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 165.0s |
| Peak memory | 6.3 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 270.1s |
| CPU sys | 8.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4297883069
This looks really cool @Dandandan -- please let me know when it is ready for review (or if you would like help, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291255155
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291134705-1700-wrcpf 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291253615
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4291134705-1702-2shzr 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (16967cd5fca63b05a3dd5785583279f877849fd6) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..16967cd5fca63b05a3dd5785583279f877849fd6) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4290105230
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4290084954)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4290084954-1689-s2g4k 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing row-group-morselization (bfa1a938d62cfa452151d2439529e05e6882c6a5) to 9a1ed57 (merge-base) [diff](https://github.com/apache/datafusion/compare/9a1ed57859398e36bb09b0af07ac20422f86b151..bfa1a938d62cfa452151d2439529e05e6882c6a5) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(parquet): row-group morselization for sibling FileStream stealing [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21766:
URL: https://github.com/apache/datafusion/pull/21766#issuecomment-4291351475
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21766#issuecomment-4291134705)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and row-group-morselization
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row-group-morselization ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 6.82 / 7.34 ±0.74 / 8.78 ms │ 6.92 / 7.35 ±0.83 / 9.02 ms │ no change │
│ QQuery 2 │ 148.66 / 149.60 ±0.64 / 150.35 ms │ 143.32 / 143.73 ±0.23 / 143.98 ms │ no change │
│ QQuery 3 │ 115.44 / 116.23 ±0.70 / 117.07 ms │ 113.22 / 113.46 ±0.27 / 113.96 ms │ no change │
│ QQuery 4 │ 1329.04 / 1422.20 ±49.48 / 1467.94 ms │ 1315.95 / 1323.65 ±8.29 / 1338.75 ms │ +1.07x faster │
│ QQuery 5 │ 172.16 / 173.81 ±2.16 / 177.98 ms │ 172.87 / 173.82 ±0.56 / 174.47 ms │ no change │
│ QQuery 6 │ 813.94 / 859.07 ±23.99 / 876.13 ms │ 866.53 / 901.96 ±21.50 / 927.25 ms │ no change │
│ QQuery 7 │ 332.86 / 346.53 ±7.57 / 354.62 ms │ 339.59 / 344.78 ±4.04 / 351.55 ms │ no change │
│ QQuery 8 │ 115.83 / 116.68 ±0.72 / 117.79 ms │ 114.22 / 115.96 ±1.95 / 119.66 ms │ no change │
│ QQuery 9 │ 102.91 / 107.79 ±2.68 / 110.59 ms │ 101.46 / 108.07 ±5.11 / 116.62 ms │ no change │
│ QQuery 10 │ 105.04 / 105.54 ±0.45 / 106.26 ms │ 102.25 / 102.72 ±0.38 / 103.41 ms │ no change │
│ QQuery 11 │ 872.96 / 929.08 ±66.00 / 1056.78 ms │ 919.37 / 959.91 ±44.08 / 1037.01 ms │ no change │
│ QQuery 12 │ 43.34 / 43.62 ±0.16 / 43.78 ms │ 46.30 / 46.58 ±0.26 / 47.08 ms │ 1.07x slower │
│ QQuery 13 │ 385.94 / 389.14 ±2.83 / 394.27 ms │ 407.11 / 411.56 ±2.55 / 414.38 ms │ 1.06x slower │
│ QQuery 14 │ 977.34 / 989.30 ±8.97 / 1004.28 ms │ 992.87 / 1009.09 ±10.97 / 1023.50 ms │ no change │
│ QQuery 15 │ 15.76 / 15.94 ±0.14 / 16.10 ms │ 14.73 / 15.28 ±0.56 / 16.31 ms │ no change │
│ QQuery 16 │ 7.87 / 8.01 ±0.12 / 8.19 ms │ 7.29 / 7.45 ±0.17 / 7.77 ms │ +1.08x faster │
│ QQuery 17 │ 232.43 / 233.70 ±1.07 / 235.43 ms │ 222.41 / 224.63 ±1.43 / 226.67 ms │ no change │
│ QQuery 18 │ 127.95 / 129.77 ±2.72 / 135.12 ms │ 122.39 / 123.19 ±0.63 / 123.83 ms │ +1.05x faster │
│ QQuery 19 │ 163.63 / 167.15 ±2.47 / 170.95 ms │ 152.87 / 154.93 ±2.30 / 159.22 ms │ +1.08x faster │
│ QQuery 20 │ 13.98 / 14.16 ±0.17 / 14.37 ms │ 13.26 / 13.50 ±0.19 / 13.81 ms │ no change │
│ QQuery 21 │ 20.28 / 20.42 ±0.18 / 20.77 ms │ 19.26 / 19.51 ±0.18 / 19.80 ms │ no change │
│ QQuery 22 │ 471.61 / 500.47 ±17.53 / 520.99 ms │ 481.95 / 490.43 ±15.52 / 521.43 ms │ no change │
│ QQuery 23 │ 813.06 / 824.55 ±8.80 / 838.05 ms │ 854.21 / 908.91 ±27.73 / 930.01 ms │ 1.10x slower │
│ QQuery 24 │ 374.31 / 380.02 ±5.35 / 389.86 ms │ 372.77 / 377.99 ±7.26 / 392.34 ms │ no change │
│ QQuery 25 │ 332.67 / 336.37 ±2.45 / 340.25 ms │ 331.23 / 333.47 ±1.67 / 336.21 ms │ no change │
│ QQuery 26 │ 77.56 / 78.78 ±2.01 / 82.78 ms │ 76.80 / 78.24 ±1.45 / 80.77 ms │ no change │
│ QQuery 27 │ 6.88 / 7.00 ±0.12 / 7.23 ms │ 6.93 / 7.08 ±0.16 / 7.39 ms │ no change │
│ QQuery 28 │ 148.32 / 149.01 ±0.56 / 149.92 ms │ 148.92 / 150.62 ±1.63 / 153.60 ms │ no change │
│ QQuery 29 │ 272.18 / 275.92 ±5.97 / 287.82 ms │ 269.89 / 274.45 ±4.57 / 282.82 ms │ no change │
│ QQuery 30 │ 41.07 / 41.11 ±0.03 / 41.15 ms │ 40.71 / 41.07 ±0.19 / 41.27 ms │ no change │
│ QQuery 31 │ 162.30 / 165.51 ±2.36 / 168.85 ms │ 163.91 / 165.16 ±1.25 / 167.27 ms │ no change │
│ QQuery 32 │ 12.94 / 13.54 ±0.37 / 14.08 ms │ 12.87 / 13.29 ±0.24 / 13.58 ms │ no change │
│ QQuery 33 │ 137.16 / 139.48 ±1.81 / 142.35 ms │ 138.24 / 139.49 ±1.10 / 141.18 ms │ no change │
│ QQuery 34 │ 6.74 / 6.95 ±0.21 / 7.35 ms │ 6.95 / 7.13 ±0.15 / 7.38 ms │ no change │
│ QQuery 35 │ 99.28 / 100.45 ±1.20 / 102.56 ms │ 101.27 / 102.18 ±0.62 / 103.14 ms │ no change │
│ QQuery 36 │ 6.87 / 6.92 ±0.04 / 6.97 ms │ 6.62 / 6.89 ±0.15 / 7.02 ms │ no change │
│ QQuery 37 │ 8.09 / 8.21 ±0.07 / 8.28 ms │ 8.10 / 8.20 ±0.10 / 8.39 ms │ no change │
│ QQuery 38 │ 85.10 / 86.42 ±1.16 / 88.33 ms │ 85.50 / 87.86 ±1.45 / 89.41 ms │ no change │
│ QQuery 39 │ 117.21 / 118.28 ±0.70 / 119.37 ms │ 116.27 / 118.54 ±1.46 / 120.27 ms │ no change │
│ QQuery 40 │ 102.68 / 105.44 ±3.34 / 111.42 ms │ 102.45 / 108.61 ±6.28 / 119.42 ms │ no change │
│ QQuery 41 │ 13.96 / 14.26 ±0.44 / 15.13 ms │ 15.10 / 15.23 ±0.13 / 15.44 ms │ 1.07x slower │
│ QQuery 42 │ 105.78 / 107.40 ±2.53 / 112.42 ms │ 109.47 / 112.60 ±4.06 / 120.57 ms │ no change │
│ QQuery 43 │ 5.57 / 5.73 ±0.22 / 6.17 ms │ 6.18 / 6.37 ±0.13 / 6.57 ms │ 1.11x slower │
│ QQuery 44 │ 11.30 / 11.42 ±0.07 / 11.48 ms │ 12.87 / 12.96 ±0.09 / 13.10 ms │ 1.13x slower │
│ QQuery 45 │ 48.63 / 49.41 ±0.78 / 50.89 ms │ 50.74 / 51.31 ±0.42 / 51.91 ms │ no change │
│ QQuery 46 │ 8.28 / 8.47 ±0.22 / 8.90 ms │ 9.12 / 9.28 ±0.14 / 9.54 ms │ 1.09x slower │
│ QQuery 47 │ 728.81 / 819.29 ±45.82 / 846.89 ms │ 816.36 / 847.00 ±16.34 / 859.58 ms │ no change │
│ QQuery 48 │ 284.98 / 290.25 ±4.05 / 294.59 ms │ 272.37 / 276.11 ±2.92 / 279.58 ms │ no change │
│ QQuery 49 │ 250.11 / 252.79 ±2.47 / 256.48 ms │ 249.63 / 251.47 ±1.17 / 253.24 ms │ no change │
│ QQuery 50 │ 201.17 / 209.37 ±7.20 / 222.16 ms │ 200.25 / 203.24 ±2.75 / 207.97 ms │ no change │
│ QQuery 51 │ 175.57 / 179.12 ±3.09 / 184.60 ms │ 175.34 / 177.09 ±1.27 / 179.21 ms │ no change │
│ QQuery 52 │ 106.64 / 107.10 ±0.37 / 107.66 ms │ 105.52 / 105.77 ±0.23 / 106.07 ms │ no change │
│ QQuery 53 │ 101.80 / 103.04 ±1.67 / 106.32 ms │ 101.17 / 101.92 ±0.92 / 103.40 ms │ no change │
│ QQuery 54 │ 141.46 / 142.71 ±0.93 / 143.54 ms │ 142.57 / 144.15 ±2.22 / 148.52 ms │ no change │
│ QQuery 55 │ 104.84 / 106.06 ±1.34 / 108.63 ms │ 105.38 / 106.14 ±0.52 / 106.81 ms │ no change │
│ QQuery 56 │ 139.36 / 144.14 ±3.17 / 148.33 ms │ 138.17 / 139.05 ±0.58 / 139.82 ms │ no change │
│ QQuery 57 │ 170.61 / 172.51 ±1.50 / 174.43 ms │ 163.22 / 164.53 ±1.29 / 166.52 ms │ no change │
│ QQuery 58 │ 317.08 / 318.08 ±0.82 / 319.26 ms │ 307.52 / 308.95 ±1.66 / 311.60 ms │ no change │
│ QQuery 59 │ 204.97 / 206.87 ±1.67 / 209.78 ms │ 192.82 / 196.08 ±2.46 / 199.59 ms │ +1.06x faster │
│ QQuery 60 │ 145.12 / 147.44 ±2.39 / 151.91 ms │ 139.90 / 141.32 ±1.24 / 143.13 ms │ no change │
│ QQuery 61 │ 14.12 / 14.23 ±0.13 / 14.49 ms │ 13.29 / 13.46 ±0.16 / 13.74 ms │ +1.06x faster │
│ QQuery 62 │ 862.81 / 892.11 ±27.58 / 939.48 ms │ 866.86 / 881.08 ±20.56 / 919.82 ms │ no change │
│ QQuery 63 │ 101.60 / 103.05 ±0.98 / 104.57 ms │ 103.89 / 105.07 ±1.28 / 107.40 ms │ no change │
│ QQuery 64 │ 659.11 / 681.32 ±20.55 / 708.00 ms │ 696.28 / 704.81 ±5.69 / 712.04 ms │ no change │
│ QQuery 65 │ 272.73 / 276.42 ±2.36 / 279.38 ms │ 245.86 / 257.99 ±9.13 / 267.19 ms │ +1.07x faster │
│ QQuery 66 │ 227.95 / 235.80 ±7.12 / 247.78 ms │ 211.98 / 218.50 ±8.35 / 234.60 ms │ +1.08x faster │
│ QQuery 67 │ 299.55 / 303.25 ±4.14 / 310.44 ms │ 287.16 / 294.77 ±11.60 / 317.85 ms │ no change │
│ QQuery 68 │ 8.48 / 8.65 ±0.24 / 9.12 ms │ 8.28 / 8.52 ±0.23 / 8.95 ms │ no change │
│ QQuery 69 │ 96.00 / 97.73 ±2.23 / 102.13 ms │ 96.45 / 97.84 ±2.22 / 102.20 ms │ no change │
│ QQuery 70 │ 312.52 / 318.54 ±4.70 / 324.32 ms │ 309.07 / 326.25 ±15.47 / 351.28 ms │ no change │
│ QQuery 71 │ 131.87 / 133.31 ±1.27 / 135.66 ms │ 135.30 / 137.82 ±2.53 / 142.36 ms │ no change │
│ QQuery 72 │ 582.22 / 592.01 ±7.59 / 601.92 ms │ 623.05 / 627.82 ±2.72 / 630.50 ms │ 1.06x slower │
│ QQuery 73 │ 6.62 / 6.78 ±0.16 / 7.06 ms │ 7.23 / 7.35 ±0.13 / 7.58 ms │ 1.08x slower │
│ QQuery 74 │ 530.60 / 542.72 ±9.71 / 553.57 ms │ 560.39 / 634.47 ±42.13 / 674.60 ms │ 1.17x slower │
│ QQuery 75 │ 265.75 / 269.17 ±2.80 / 273.29 ms │ 264.22 / 268.90 ±4.39 / 277.10 ms │ no change │
│ QQuery 76 │ 129.68 / 130.75 ±1.35 / 133.27 ms │ 128.85 / 129.76 ±0.81 / 131.14 ms │ no change │
│ QQuery 77 │ 190.80 / 191.79 ±1.28 / 194.30 ms │ 186.32 / 187.11 ±1.10 / 189.20 ms │ no change │
│ QQuery 78 │ 349.04 / 349.85 ±0.81 / 350.98 ms │ 324.34 / 328.43 ±3.25 / 334.04 ms │ +1.07x faster │
│ QQuery 79 │ 255.75 / 258.45 ±3.50 / 265.38 ms │ 225.69 / 226.21 ±0.45 / 226.95 ms │ +1.14x faster │
│ QQuery 80 │ 320.91 / 323.97 ±2.19 / 327.24 ms │ 320.66 / 324.25 ±2.85 / 329.41 ms │ no change │
│ QQuery 81 │ 26.42 / 26.71 ±0.25 / 27.01 ms │ 27.34 / 27.65 ±0.25 / 28.10 ms │ no change │
│ QQuery 82 │ 38.54 / 39.07 ±0.50 / 39.77 ms │ 40.71 / 41.83 ±0.81 / 43.09 ms │ 1.07x slower │
│ QQuery 83 │ 36.39 / 36.79 ±0.38 / 37.50 ms │ 38.26 / 38.77 ±0.51 / 39.74 ms │ 1.05x slower │
│ QQuery 84 │ 45.89 / 46.06 ±0.13 / 46.21 ms │ 47.59 / 47.75 ±0.22 / 48.18 ms │ no change │
│ QQuery 85 │ 138.86 / 139.88 ±0.98 / 141.74 ms │ 144.02 / 145.38 ±1.54 / 148.26 ms │ no change │
│ QQuery 86 │ 36.86 / 37.29 ±0.33 / 37.73 ms │ 38.06 / 38.61 ±0.43 / 39.28 ms │ no change │
│ QQuery 87 │ 3.40 / 3.51 ±0.16 / 3.83 ms │ 3.76 / 3.88 ±0.14 / 4.13 ms │ 1.10x slower │
│ QQuery 88 │ 98.65 / 99.97 ±1.07 / 101.89 ms │ 102.14 / 104.31 ±1.78 / 106.49 ms │ no change │
│ QQuery 89 │ 115.71 / 118.80 ±5.21 / 129.18 ms │ 120.04 / 120.42 ±0.36 / 120.95 ms │ no change │
│ QQuery 90 │ 21.58 / 22.33 ±0.56 / 23.16 ms │ 23.35 / 24.05 ±0.78 / 25.36 ms │ 1.08x slower │
│ QQuery 91 │ 57.05 / 58.20 ±0.84 / 59.40 ms │ 58.05 / 59.73 ±1.50 / 61.76 ms │ no change │
│ QQuery 92 │ 55.37 / 55.98 ±0.43 / 56.63 ms │ 55.42 / 55.81 ±0.58 / 56.97 ms │ no change │
│ QQuery 93 │ 178.28 / 180.31 ±1.30 / 182.14 ms │ 178.19 / 182.38 ±3.14 / 186.66 ms │ no change │
│ QQuery 94 │ 59.89 / 60.75 ±0.97 / 62.56 ms │ 59.34 / 59.84 ±0.33 / 60.22 ms │ no change │
│ QQuery 95 │ 123.54 / 124.64 ±0.76 / 125.93 ms │ 124.50 / 125.60 ±0.97 / 127.37 ms │ no change │
│ QQuery 96 │ 68.15 / 71.28 ±3.39 / 77.85 ms │ 66.92 / 67.10 ±0.18 / 67.44 ms │ +1.06x faster │
│ QQuery 97 │ 116.23 / 118.10 ±1.24 / 119.58 ms │ 115.34 / 115.92 ±0.49 / 116.77 ms │ no change │
│ QQuery 98 │ 150.22 / 151.96 ±2.03 / 155.38 ms │ 147.86 / 149.79 ±1.10 / 151.10 ms │ no change │
│ QQuery 99 │ 10775.28 / 11017.01 ±136.34 / 11196.90 ms │ 10809.31 / 11003.35 ±115.32 / 11146.77 ms │ no change │
└───────────┴───────────────────────────────────────────┴───────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 31223.50ms │
│ Total Time (row-group-morselization) │ 31303.93ms │
│ Average Time (HEAD) │ 315.39ms │
│ Average Time (row-group-morselization) │ 316.20ms │
│ Queries Faster │ 11 │
│ Queries Slower │ 14 │
│ Queries with No Change │ 74 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 259.0s |
| CPU sys | 8.7s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 264.1s |
| CPU sys | 8.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org